Sat Mar 5 03:31:41 UTC 2022 Starting regionserver on cvp328.sjc.aristanetworks.com core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 79745 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 65536 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 79745 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited 2022-03-05 03:31:43,068 INFO [main] regionserver.HRegionServer: STARTING executorService HRegionServer 2022-03-05 03:31:43,069 INFO [main] util.VersionInfo: HBase 2.4.8 2022-03-05 03:31:43,069 INFO [main] util.VersionInfo: Source code repository git://buildkitsandbox/hbase-src revision=cb19adc2e3254e1e67bc5509e436fb407959232f 2022-03-05 03:31:43,069 INFO [main] util.VersionInfo: Compiled by root on Sat Nov 6 00:59:03 UTC 2021 2022-03-05 03:31:43,069 INFO [main] util.VersionInfo: From source with checksum 6bd029f558ce15696daeaa252cc84fc34376feb5ae5f3cfeb21ba8313b6eae624a3d13f771c2e661025d17c04a89fbbb05c36baa9680b429ab9e268ce3d05366 2022-03-05 03:31:43,288 INFO [main] util.ServerCommandLine: hbase.tmp.dir: /data/hbase 2022-03-05 03:31:43,288 INFO [main] util.ServerCommandLine: hbase.rootdir: hdfs://mycluster/hbase 2022-03-05 03:31:43,288 INFO [main] util.ServerCommandLine: hbase.cluster.distributed: true 2022-03-05 03:31:43,288 INFO [main] util.ServerCommandLine: hbase.zookeeper.quorum: cvp328.sjc.aristanetworks.com,cvp365.sjc.aristanetworks.com,cvp90.sjc.aristanetworks.com 2022-03-05 03:31:43,289 INFO [main] util.ServerCommandLine: env:PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/cvpi/bin:/cvpi/jdk/bin:/cvpi/zookeeper/bin:/cvpi/clickhouse/bin:/cvpi/hadoop/bin:/cvpi/hadoop/sbin:/cvpi/hbase/bin 2022-03-05 03:31:43,289 INFO [main] util.ServerCommandLine: env:HADOOP_CONF_DIR=/cvpi/hadoop/etc/hadoop 2022-03-05 03:31:43,289 INFO [main] util.ServerCommandLine: env:LOAD_BALANCER_IP= 2022-03-05 03:31:43,289 INFO [main] util.ServerCommandLine: env:HBASE_PID_DIR=/var/run/cvpi/hbase/pid 2022-03-05 03:31:43,289 INFO [main] util.ServerCommandLine: env:KUBECONFIG=/etc/kubernetes/kubeconfig.yaml 2022-03-05 03:31:43,289 INFO [main] util.ServerCommandLine: env:CVP_HOME=/cvpi 2022-03-05 03:31:43,289 INFO [main] util.ServerCommandLine: env:HBASE_REGIONSERVER_OPTS= -javaagent:/cvpi/jdk/lib/jmx_prometheus_javaagent.jar=7073:/cvpi/hbase/conf/hbase-jmx-exporter-config.yml 2022-03-05 03:31:43,289 INFO [main] util.ServerCommandLine: env:TERTIARY_HOSTNAME=cvp90.sjc.aristanetworks.com 2022-03-05 03:31:43,289 INFO [main] util.ServerCommandLine: env:CVP_VERSION=2022.1.0 2022-03-05 03:31:43,289 INFO [main] util.ServerCommandLine: env:DEPLOYMENT_MODEL=DEFAULT 2022-03-05 03:31:43,289 INFO [main] util.ServerCommandLine: env:LD_LIBRARY_PATH=:/cvpi/hadoop/lib/native:/cvpi/hadoop/lib/native 2022-03-05 03:31:43,289 INFO [main] util.ServerCommandLine: env:LOGNAME=cvp 2022-03-05 03:31:43,289 INFO [main] util.ServerCommandLine: env:LOAD_BALANCER_PORT= 2022-03-05 03:31:43,289 INFO [main] util.ServerCommandLine: env:JVM_PID=13115 2022-03-05 03:31:43,289 INFO [main] util.ServerCommandLine: env:HBASE_REST_OPTS= 2022-03-05 03:31:43,289 INFO [main] util.ServerCommandLine: env:PWD=/home/cvp 2022-03-05 03:31:43,289 INFO [main] util.ServerCommandLine: env:CVP_DATA_DIR=/data 2022-03-05 03:31:43,289 INFO [main] util.ServerCommandLine: env:CVP_USER=cvp 2022-03-05 03:31:43,290 INFO [main] util.ServerCommandLine: env:CV_WIFI_HA_CLUSTER_IP= 2022-03-05 03:31:43,290 INFO [main] util.ServerCommandLine: env:HBASE_ROOT_LOGGER=INFO,RFA 2022-03-05 03:31:43,290 INFO [main] util.ServerCommandLine: env:SHELL=/bin/bash 2022-03-05 03:31:43,290 INFO [main] util.ServerCommandLine: env:HBASE_HEAPSIZE=1536 2022-03-05 03:31:43,290 INFO [main] util.ServerCommandLine: env:HBASE_ENV_INIT=true 2022-03-05 03:31:43,290 INFO [main] util.ServerCommandLine: env:CVP_INTERFACE=eth0 2022-03-05 03:31:43,290 INFO [main] util.ServerCommandLine: env:SECONDARY_HOST_IP=172.30.41.155 2022-03-05 03:31:43,290 INFO [main] util.ServerCommandLine: env:HBASE_MASTER_OPTS= -javaagent:/cvpi/jdk/lib/jmx_prometheus_javaagent.jar=7072:/cvpi/hbase/conf/hbase-jmx-exporter-config.yml 2022-03-05 03:31:43,290 INFO [main] util.ServerCommandLine: env:HBASE_MANAGES_ZK=false 2022-03-05 03:31:43,290 INFO [main] util.ServerCommandLine: env:SED=sed 2022-03-05 03:31:43,290 INFO [main] util.ServerCommandLine: env:HADOOP_HOME=/cvpi/hadoop 2022-03-05 03:31:43,290 INFO [main] util.ServerCommandLine: env:HBASE_NICENESS=0 2022-03-05 03:31:43,290 INFO [main] util.ServerCommandLine: env:HBASE_OPTS=-XX:+UseG1GC -XX:MaxGCPauseMillis=100 -XX:+ParallelRefProcEnabled -XX:+PerfDisableSharedMem -Djava.net.preferIPv4Stack=true -Dlog4j.appender.RFA.MaxBackupIndex=0 -Dlog4j.appender.RFAS.MaxBackupIndex=0 -Djava.util.logging.config.class=org.apache.hadoop.hbase.logging.JulToSlf4jInitializer -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/cvpi/hbase/logs/hbase-cvp-regionserver-cvp328.sjc.aristanetworks.com.gc -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=100M -javaagent:/cvpi/jdk/lib/jmx_prometheus_javaagent.jar=7073:/cvpi/hbase/conf/hbase-jmx-exporter-config.yml -Dhbase.log.dir=/cvpi/hbase/logs -Dhbase.log.file=hbase-cvp-regionserver-cvp328.sjc.aristanetworks.com.log -Dhbase.home.dir=/cvpi/hbase -Dhbase.id.str=cvp -Dhbase.root.logger=INFO,RFA -Djava.library.path=/cvpi/hadoop/lib/native:/cvpi/hadoop/lib/native -Dhbase.security.logger=INFO,RFAS 2022-03-05 03:31:43,290 INFO [main] util.ServerCommandLine: env:HBASE_SECURITY_LOGGER=INFO,RFAS 2022-03-05 03:31:43,290 INFO [main] util.ServerCommandLine: env:PRIMARY_DEVICE_INTF_IP=172.30.41.118 2022-03-05 03:31:43,290 INFO [main] util.ServerCommandLine: env:SHLVL=3 2022-03-05 03:31:43,290 INFO [main] util.ServerCommandLine: env:HBASE_LOGFILE=hbase-cvp-regionserver-cvp328.sjc.aristanetworks.com.log 2022-03-05 03:31:43,290 INFO [main] util.ServerCommandLine: env:CVP_LOG_LEVEL=INFO 2022-03-05 03:31:43,290 INFO [main] util.ServerCommandLine: env:TERTIARY_DEVICE_INTF_IP=172.31.0.219 2022-03-05 03:31:43,290 INFO [main] util.ServerCommandLine: env:JAVA_HOME=/cvpi/jdk 2022-03-05 03:31:43,291 INFO [main] util.ServerCommandLine: env:KUBE_CLUSTER_NETWORK=100.96.0.0/16 2022-03-05 03:31:43,291 INFO [main] util.ServerCommandLine: env:HBASE_STOP_TIMEOUT=250 2022-03-05 03:31:43,291 INFO [main] util.ServerCommandLine: env:CVP_MODE=multinode 2022-03-05 03:31:43,291 INFO [main] util.ServerCommandLine: env:LANG=en_US.UTF-8 2022-03-05 03:31:43,291 INFO [main] util.ServerCommandLine: env:KUBE_CLUSTER_DNS=100.96.128.2 2022-03-05 03:31:43,291 INFO [main] util.ServerCommandLine: env:SECONDARY_HOSTNAME=cvp365.sjc.aristanetworks.com 2022-03-05 03:31:43,291 INFO [main] util.ServerCommandLine: env:TERTIARY_HOST_IP=172.31.0.219 2022-03-05 03:31:43,291 INFO [main] util.ServerCommandLine: env:PRIMARY_HOST_IP=172.30.41.118 2022-03-05 03:31:43,291 INFO [main] util.ServerCommandLine: env:HBASE_CLASSPATH=:/cvpi/hadoop/etc/hadoop 2022-03-05 03:31:43,291 INFO [main] util.ServerCommandLine: env:KUBE_SERVICE_NETWORK=100.96.128.0/17 2022-03-05 03:31:43,291 INFO [main] util.ServerCommandLine: env:CV_WIFI_ENABLED= 2022-03-05 03:31:43,291 INFO [main] util.ServerCommandLine: env:SECONDARY_DEVICE_INTF_IP=172.30.41.155 2022-03-05 03:31:43,291 INFO [main] util.ServerCommandLine: env:PRIMARY_HOSTNAME=cvp328.sjc.aristanetworks.com 2022-03-05 03:31:43,291 INFO [main] util.ServerCommandLine: env:HBASE_SERVER_JAAS_OPTS= 2022-03-05 03:31:43,291 INFO [main] util.ServerCommandLine: env:HBASE_IDENT_STRING=cvp 2022-03-05 03:31:43,291 INFO [main] util.ServerCommandLine: env:HBASE_ZNODE_FILE=/var/run/cvpi/hbase/pid/hbase-cvp-regionserver.znode 2022-03-05 03:31:43,291 INFO [main] util.ServerCommandLine: env:HBASE_LOG_PREFIX=hbase-cvp-regionserver-cvp328.sjc.aristanetworks.com 2022-03-05 03:31:43,291 INFO [main] util.ServerCommandLine: env:HBASE_LOG_DIR=/cvpi/hbase/logs 2022-03-05 03:31:43,291 INFO [main] util.ServerCommandLine: env:USER=cvp 2022-03-05 03:31:43,291 INFO [main] util.ServerCommandLine: /cvpi/hbase/lib/woodstox-core-5.0.3.jar:/cvpi/hbase/lib/zookeeper-3.5.7.jar:/cvpi/hbase/lib/zookeeper-jute-3.5.7.jar:/cvpi/hbase/lib/client-facing-thirdparty/audience-annotations-0.5.0.jar:/cvpi/hbase/lib/client-facing-thirdparty/commons-logging-1.2.jar:/cvpi/hbase/lib/client-facing-thirdparty/htrace-core4-4.2.0-incubating.jar:/cvpi/hbase/lib/client-facing-thirdparty/jcl-over-slf4j-1.7.30.jar:/cvpi/hbase/lib/client-facing-thirdparty/jul-to-slf4j-1.7.30.jar:/cvpi/hbase/lib/client-facing-thirdparty/log4j-1.2.17.jar:/cvpi/hbase/lib/client-facing-thirdparty/slf4j-api-1.7.30.jar:/cvpi/hadoop/etc/hadoop:/cvpi/hadoop/share/hadoop/common/lib/*:/cvpi/hadoop/share/hadoop/common/*:/cvpi/hadoop/share/hadoop/hdfs:/cvpi/hadoop/share/hadoop/hdfs/lib/*:/cvpi/hadoop/share/hadoop/hdfs/*:/cvpi/hadoop/share/hadoop/mapreduce/lib/*:/cvpi/hadoop/share/hadoop/mapreduce/*:/cvpi/hadoop/share/hadoop/yarn/lib/*:/cvpi/hadoop/share/hadoop/yarn/*:/cvpi/hadoop/contrib/capacity-scheduler/*.jar::/cvpi/hadoop/etc/hadoop 2022-03-05 03:31:43,292 INFO [main] util.ServerCommandLine: env:AERIS_INGEST_KEY=aeris_magic_29932 2022-03-05 03:31:43,292 INFO [main] util.ServerCommandLine: env:SERVER_GC_OPTS=-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/cvpi/hbase/logs/hbase-cvp-regionserver-cvp328.sjc.aristanetworks.com.gc -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=100M 2022-03-05 03:31:43,292 INFO [main] util.ServerCommandLine: env:KUBE_POD_NETWORK=100.96.0.0/17 2022-03-05 03:31:43,292 INFO [main] util.ServerCommandLine: env:HBASE_LIBRARY_PATH=/cvpi/hadoop/lib/native 2022-03-05 03:31:43,292 INFO [main] util.ServerCommandLine: env:CVP_GROUP=cvp 2022-03-05 03:31:43,292 INFO [main] util.ServerCommandLine: env:HBASE_AUTOSTART_FILE=/var/run/cvpi/hbase/pid/hbase-cvp-regionserver.autostart 2022-03-05 03:31:43,292 INFO [main] util.ServerCommandLine: env:GREP=grep 2022-03-05 03:31:43,292 INFO [main] util.ServerCommandLine: env:HBASE_THRIFT_OPTS= 2022-03-05 03:31:43,292 INFO [main] util.ServerCommandLine: env:HBASE_HOME=/cvpi/hbase 2022-03-05 03:31:43,292 INFO [main] util.ServerCommandLine: env:HOME=/home/cvp 2022-03-05 03:31:43,294 INFO [main] util.ServerCommandLine: env:MALLOC_ARENA_MAX=4 2022-03-05 03:31:43,294 INFO [main] util.ServerCommandLine: vmName=OpenJDK 64-Bit Server VM, vmVendor=AdoptOpenJDK, vmVersion=25.265-b01 2022-03-05 03:31:43,294 INFO [main] util.ServerCommandLine: vmInputArguments=[-Dproc_regionserver, -XX:OnOutOfMemoryError=kill -9 %p, -Xmx1536m, -XX:+UseG1GC, -XX:MaxGCPauseMillis=100, -XX:+ParallelRefProcEnabled, -XX:+PerfDisableSharedMem, -Djava.net.preferIPv4Stack=true, -Dlog4j.appender.RFA.MaxBackupIndex=0, -Dlog4j.appender.RFAS.MaxBackupIndex=0, -Djava.util.logging.config.class=org.apache.hadoop.hbase.logging.JulToSlf4jInitializer, -verbose:gc, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -Xloggc:/cvpi/hbase/logs/hbase-cvp-regionserver-cvp328.sjc.aristanetworks.com.gc, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=5, -XX:GCLogFileSize=100M, -javaagent:/cvpi/jdk/lib/jmx_prometheus_javaagent.jar=7073:/cvpi/hbase/conf/hbase-jmx-exporter-config.yml, -Dhbase.log.dir=/cvpi/hbase/logs, -Dhbase.log.file=hbase-cvp-regionserver-cvp328.sjc.aristanetworks.com.log, -Dhbase.home.dir=/cvpi/hbase, -Dhbase.id.str=cvp, -Dhbase.root.logger=INFO,RFA, -Djava.library.path=/cvpi/hadoop/lib/native:/cvpi/hadoop/lib/native, -Dhbase.security.logger=INFO,RFAS] 2022-03-05 03:31:43,698 INFO [main] metrics.MetricRegistries: Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2022-03-05 03:31:43,817 WARN [main] util.FSUtils: Configuration "dfs.client.read.shortcircuit.skip.checksum" should not be set to true. HBase checksum doesn't require it, see https://issues.apache.org/jira/browse/HBASE-6868. 2022-03-05 03:31:43,990 INFO [main] regionserver.RSRpcServices: regionserver/cvp328:16201 server-side Connection retries=45 2022-03-05 03:31:44,006 INFO [main] ipc.RpcExecutor: Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=1024, handlerCount=16 2022-03-05 03:31:44,008 INFO [main] ipc.RpcExecutor: Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=1024, handlerCount=10 2022-03-05 03:31:44,008 INFO [main] ipc.RWQueueRpcExecutor: priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=9 scanQueues=0 scanHandlers=0 2022-03-05 03:31:44,009 INFO [main] ipc.RpcExecutor: Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=1024, handlerCount=3 2022-03-05 03:31:44,009 INFO [main] ipc.RpcExecutor: Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=1024, handlerCount=1 2022-03-05 03:31:44,165 INFO [main] ipc.RpcServerFactory: Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2022-03-05 03:31:44,203 INFO [main] Configuration.deprecation: hbase.ipc.server.reservoir.initial.buffer.size is deprecated. Instead, use hbase.server.allocator.buffer.size 2022-03-05 03:31:44,203 INFO [main] Configuration.deprecation: hbase.ipc.server.reservoir.initial.max is deprecated. Instead, use hbase.server.allocator.max.buffer.count 2022-03-05 03:31:44,291 INFO [main] ipc.NettyRpcServer: Bind to /172.30.41.118:16201 2022-03-05 03:31:44,374 INFO [main] Configuration.deprecation: hbase.offheapcache.minblocksize is deprecated. Instead, use hbase.blockcache.minblocksize 2022-03-05 03:31:44,376 INFO [main] hfile.BlockCacheFactory: Allocating BlockCache size=307.20 MB, blockSize=64 KB 2022-03-05 03:31:45,035 INFO [main] fs.HFileSystem: Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2022-03-05 03:31:45,039 INFO [main] fs.HFileSystem: Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2022-03-05 03:31:45,056 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=regionserver:16201 connecting to ZooKeeper ensemble=cvp328.sjc.aristanetworks.com:2181,cvp365.sjc.aristanetworks.com:2181,cvp90.sjc.aristanetworks.com:2181 2022-03-05 03:31:45,063 INFO [main] zookeeper.ZooKeeper: Client environment:zookeeper.version=3.5.7-f0fdd52973d373ffd9c86b81d99842dc2c7f660e, built on 02/10/2020 11:30 GMT 2022-03-05 03:31:45,063 INFO [main] zookeeper.ZooKeeper: Client environment:host.name=cvp328.sjc.aristanetworks.com 2022-03-05 03:31:45,064 INFO [main] zookeeper.ZooKeeper: Client environment:java.version=1.8.0_265 2022-03-05 03:31:45,064 INFO [main] zookeeper.ZooKeeper: Client environment:java.vendor=AdoptOpenJDK 2022-03-05 03:31:45,064 INFO [main] zookeeper.ZooKeeper: Client environment:java.home=/cvpi/jdk/jre 2022-03-05 03:31:45,064 INFO [main] zookeeper.ZooKeeper: op/yarn/hadoop-yarn-api-3.1.4.jar:/cvpi/hadoop/share/hadoop/yarn/hadoop-yarn-server-nodemanager-3.1.4.jar:/cvpi/hadoop/share/hadoop/yarn/hadoop-yarn-common-3.1.4.jar:/cvpi/hadoop/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.4.jar:/cvpi/hadoop/share/hadoop/yarn/hadoop-yarn-client-3.1.4.jar:/cvpi/hadoop/share/hadoop/yarn/hadoop-yarn-server-common-3.1.4.jar:/cvpi/hadoop/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-3.1.4.jar:/cvpi/hadoop/share/hadoop/yarn/hadoop-yarn-server-tests-3.1.4.jar:/cvpi/hadoop/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-3.1.4.jar:/cvpi/hadoop/share/hadoop/yarn/hadoop-yarn-registry-3.1.4.jar:/cvpi/hadoop/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-3.1.4.jar:/cvpi/hadoop/share/hadoop/yarn/hadoop-yarn-services-core-3.1.4.jar:/cvpi/hadoop/share/hadoop/yarn/hadoop-yarn-services-api-3.1.4.jar:/cvpi/hadoop/contrib/capacity-scheduler/*.jar::/cvpi/hadoop/etc/hadoop:/cvpi/jdk/lib/jmx_prometheus_javaagent.jar 2022-03-05 03:31:45,064 INFO [main] zookeeper.ZooKeeper: Client environment:java.library.path=/cvpi/hadoop/lib/native:/cvpi/hadoop/lib/native 2022-03-05 03:31:45,064 INFO [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp 2022-03-05 03:31:45,064 INFO [main] zookeeper.ZooKeeper: Client environment:java.compiler= 2022-03-05 03:31:45,064 INFO [main] zookeeper.ZooKeeper: Client environment:os.name=Linux 2022-03-05 03:31:45,064 INFO [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64 2022-03-05 03:31:45,064 INFO [main] zookeeper.ZooKeeper: Client environment:os.version=5.12.12-1.el7.elrepo.x86_64 2022-03-05 03:31:45,064 INFO [main] zookeeper.ZooKeeper: Client environment:user.name=cvp 2022-03-05 03:31:45,064 INFO [main] zookeeper.ZooKeeper: Client environment:user.home=/home/cvp 2022-03-05 03:31:45,064 INFO [main] zookeeper.ZooKeeper: Client environment:user.dir=/home/cvp 2022-03-05 03:31:45,064 INFO [main] zookeeper.ZooKeeper: Client environment:os.memory.free=197MB 2022-03-05 03:31:45,064 INFO [main] zookeeper.ZooKeeper: Client environment:os.memory.max=1536MB 2022-03-05 03:31:45,064 INFO [main] zookeeper.ZooKeeper: Client environment:os.memory.total=314MB 2022-03-05 03:31:45,066 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=cvp328.sjc.aristanetworks.com:2181,cvp365.sjc.aristanetworks.com:2181,cvp90.sjc.aristanetworks.com:2181 sessionTimeout=20000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@24097e9b 2022-03-05 03:31:45,070 INFO [main] common.X509Util: Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation 2022-03-05 03:31:45,074 INFO [main] zookeeper.ClientCnxnSocket: jute.maxbuffer value is 4194304 Bytes 2022-03-05 03:31:45,081 INFO [main] zookeeper.ClientCnxn: zookeeper.request.timeout value is 0. feature enabled= 2022-03-05 03:31:45,087 INFO [main-SendThread(cvp365.sjc.aristanetworks.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server cvp365.sjc.aristanetworks.com/172.30.41.155:2181. Will not attempt to authenticate using SASL (unknown error) 2022-03-05 03:31:45,091 INFO [main-SendThread(cvp365.sjc.aristanetworks.com:2181)] zookeeper.ClientCnxn: Socket connection established, initiating session, client: /172.30.41.118:49346, server: cvp365.sjc.aristanetworks.com/172.30.41.155:2181 2022-03-05 03:31:45,098 INFO [main-SendThread(cvp365.sjc.aristanetworks.com:2181)] zookeeper.ClientCnxn: Session establishment complete on server cvp365.sjc.aristanetworks.com/172.30.41.155:2181, sessionid = 0x200006e600e0021, negotiated timeout = 60000 2022-03-05 03:31:45,154 INFO [main] util.log: Logging initialized @3383ms to org.apache.hbase.thirdparty.org.eclipse.jetty.util.log.Slf4jLog 2022-03-05 03:31:45,274 INFO [main] http.HttpServer: Added global filter 'safety' (class=org.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter) 2022-03-05 03:31:45,274 INFO [main] http.HttpServer: Added global filter 'clickjackingprevention' (class=org.apache.hadoop.hbase.http.ClickjackingPreventionFilter) 2022-03-05 03:31:45,275 INFO [main] http.HttpServer: Added global filter 'securityheaders' (class=org.apache.hadoop.hbase.http.SecurityHeadersFilter) 2022-03-05 03:31:45,277 INFO [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context regionserver 2022-03-05 03:31:45,277 INFO [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs 2022-03-05 03:31:45,277 INFO [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context static 2022-03-05 03:31:45,285 INFO [main] http.HttpServer: ASYNC_PROFILER_HOME environment variable and async.profiler.home system property not specified. Disabling /prof endpoint. 2022-03-05 03:31:45,391 INFO [main] http.HttpServer: Jetty bound to port 16301 2022-03-05 03:31:45,392 INFO [main] server.Server: jetty-9.4.41.v20210516; built: 2021-05-16T23:56:28.993Z; git: 98607f93c7833e7dc59489b13f3cb0a114fb9f4c; jvm 1.8.0_265-b01 2022-03-05 03:31:45,416 INFO [main] http.SecurityHeadersFilter: Added security headers filter 2022-03-05 03:31:45,419 INFO [main] handler.ContextHandler: Started o.a.h.t.o.e.j.s.ServletContextHandler@17740dae{logs,/logs,file:///cvpi/hbase/logs/,AVAILABLE} 2022-03-05 03:31:45,419 INFO [main] http.SecurityHeadersFilter: Added security headers filter 2022-03-05 03:31:45,420 INFO [main] handler.ContextHandler: Started o.a.h.t.o.e.j.s.ServletContextHandler@4d774249{static,/static,file:///cvpi/hbase/hbase-webapps/static/,AVAILABLE} 2022-03-05 03:31:45,489 INFO [main] webapp.StandardDescriptorProcessor: NO JSP Support for /, did not find org.apache.hbase.thirdparty.org.eclipse.jetty.jsp.JettyJspServlet 2022-03-05 03:31:45,497 INFO [main] server.session: DefaultSessionIdManager workerName=node0 2022-03-05 03:31:45,497 INFO [main] server.session: No SessionScavenger set, using defaults 2022-03-05 03:31:45,498 INFO [main] server.session: node0 Scavenging every 660000ms 2022-03-05 03:31:45,503 INFO [main] http.SecurityHeadersFilter: Added security headers filter 2022-03-05 03:31:45,518 INFO [main] handler.ContextHandler: Started o.a.h.t.o.e.j.w.WebAppContext@62891fc8{regionserver,/,file:///cvpi/hbase/hbase-webapps/regionserver/,AVAILABLE}{file:/cvpi/hbase/hbase-webapps/regionserver} 2022-03-05 03:31:45,525 INFO [main] server.AbstractConnector: Started ServerConnector@289778cd{HTTP/1.1, (http/1.1)}{0.0.0.0:16301} 2022-03-05 03:31:45,525 INFO [main] server.Server: Started @3755ms 2022-03-05 03:31:47,467 INFO [regionserver/cvp328:16201] regionserver.HRegionServer: ClusterId : 92148e91-f244-4419-aaf9-953d99538486 2022-03-05 03:31:47,524 INFO [ReadOnlyZKClient-cvp328.sjc.aristanetworks.com:2181,cvp365.sjc.aristanetworks.com:2181,cvp90.sjc.aristanetworks.com:2181@0x7dafa7e3] zookeeper.ZooKeeper: Initiating client connection, connectString=cvp328.sjc.aristanetworks.com:2181,cvp365.sjc.aristanetworks.com:2181,cvp90.sjc.aristanetworks.com:2181 sessionTimeout=20000 watcher=org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$66/1382608978@1355fa69 2022-03-05 03:31:47,524 INFO [ReadOnlyZKClient-cvp328.sjc.aristanetworks.com:2181,cvp365.sjc.aristanetworks.com:2181,cvp90.sjc.aristanetworks.com:2181@0x7dafa7e3] zookeeper.ClientCnxnSocket: jute.maxbuffer value is 4194304 Bytes 2022-03-05 03:31:47,524 INFO [ReadOnlyZKClient-cvp328.sjc.aristanetworks.com:2181,cvp365.sjc.aristanetworks.com:2181,cvp90.sjc.aristanetworks.com:2181@0x7dafa7e3] zookeeper.ClientCnxn: zookeeper.request.timeout value is 0. feature enabled= 2022-03-05 03:31:47,525 INFO [ReadOnlyZKClient-cvp328.sjc.aristanetworks.com:2181,cvp365.sjc.aristanetworks.com:2181,cvp90.sjc.aristanetworks.com:2181@0x7dafa7e3-SendThread(cvp328.sjc.aristanetworks.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server cvp328.sjc.aristanetworks.com/172.30.41.118:2181. Will not attempt to authenticate using SASL (unknown error) 2022-03-05 03:31:47,537 INFO [ReadOnlyZKClient-cvp328.sjc.aristanetworks.com:2181,cvp365.sjc.aristanetworks.com:2181,cvp90.sjc.aristanetworks.com:2181@0x7dafa7e3-SendThread(cvp328.sjc.aristanetworks.com:2181)] zookeeper.ClientCnxn: Socket connection established, initiating session, client: /172.30.41.118:59124, server: cvp328.sjc.aristanetworks.com/172.30.41.118:2181 2022-03-05 03:31:47,539 INFO [ReadOnlyZKClient-cvp328.sjc.aristanetworks.com:2181,cvp365.sjc.aristanetworks.com:2181,cvp90.sjc.aristanetworks.com:2181@0x7dafa7e3-SendThread(cvp328.sjc.aristanetworks.com:2181)] zookeeper.ClientCnxn: Session establishment complete on server cvp328.sjc.aristanetworks.com/172.30.41.118:2181, sessionid = 0x10000006c9d01a4, negotiated timeout = 60000 2022-03-05 03:31:47,578 INFO [regionserver/cvp328:16201] regionserver.RegionServerCoprocessorHost: System coprocessor loading is enabled 2022-03-05 03:31:47,578 INFO [regionserver/cvp328:16201] regionserver.RegionServerCoprocessorHost: Table coprocessor loading is enabled 2022-03-05 03:31:47,584 INFO [regionserver/cvp328:16201] regionserver.HRegionServer: reportForDuty to master=cvp328.sjc.aristanetworks.com,16000,1646451103368 with isa=cvp328.sjc.aristanetworks.com/172.30.41.118:16201, startcode=1646451103340 2022-03-05 03:31:47,765 INFO [regionserver/cvp328:16201] wal.WALFactory: Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2022-03-05 03:31:47,908 INFO [regionserver/cvp328:16201] regionserver.MetricsRegionServerWrapperImpl: Computing regionserver metrics every 5000 milliseconds 2022-03-05 03:31:47,932 INFO [regionserver/cvp328:16201] regionserver.MemStoreFlusher: globalMemStoreLimit=614.4 M, globalMemStoreLimitLowMark=583.7 M, Offheap=false 2022-03-05 03:31:47,937 INFO [regionserver/cvp328:16201] throttle.PressureAwareCompactionThroughputController: Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2022-03-05 03:31:47,938 INFO [regionserver/cvp328:16201] hbase.ChoreService: Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2022-03-05 03:31:47,939 INFO [regionserver/cvp328:16201] regionserver.HRegionServer: CompactionChecker runs every PT10S 2022-03-05 03:31:47,973 INFO [regionserver/cvp328:16201] hbase.ChoreService: Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2022-03-05 03:31:47,977 INFO [regionserver/cvp328:16201] hbase.ChoreService: Chore ScheduledChore name=CompactionChecker, period=10000, unit=MILLISECONDS is enabled. 2022-03-05 03:31:47,977 INFO [regionserver/cvp328:16201] hbase.ChoreService: Chore ScheduledChore name=MemstoreFlusherChore, period=10000, unit=MILLISECONDS is enabled. 2022-03-05 03:31:47,977 INFO [regionserver/cvp328:16201] hbase.ChoreService: Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2022-03-05 03:31:47,999 INFO [regionserver/cvp328:16201] regionserver.HeapMemoryManager: Starting, tuneOn=false 2022-03-05 03:31:48,002 INFO [regionserver/cvp328:16201] hbase.ChoreService: Chore ScheduledChore name=cvp328.sjc.aristanetworks.com,16201,1646451103340-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2022-03-05 03:31:48,004 INFO [regionserver/cvp328:16201] regionserver.ChunkCreator: Allocating data MemStoreChunkPool with chunk size 2 MB, max count 276, initial count 0 2022-03-05 03:31:48,005 INFO [regionserver/cvp328:16201] regionserver.ChunkCreator: Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 307, initial count 0 2022-03-05 03:31:48,026 INFO [regionserver/cvp328:16201] regionserver.Replication: cvp328.sjc.aristanetworks.com,16201,1646451103340 started 2022-03-05 03:31:48,026 INFO [regionserver/cvp328:16201] regionserver.HRegionServer: Serving as cvp328.sjc.aristanetworks.com,16201,1646451103340, RpcServer on cvp328.sjc.aristanetworks.com/172.30.41.118:16201, sessionid=0x200006e600e0021 2022-03-05 03:31:48,031 INFO [regionserver/cvp328:16201] quotas.RegionServerRpcQuotaManager: Quota support disabled 2022-03-05 03:31:48,031 INFO [regionserver/cvp328:16201] quotas.RegionServerSpaceQuotaManager: Quota support disabled, not starting space quota manager. 2022-03-05 03:31:53,060 INFO [regionserver/cvp328:16201] wal.AbstractFSWAL: WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340, suffix=, logDir=hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340, archiveDir=hdfs://mycluster/hbase/oldWALs, maxLogs=32 2022-03-05 03:31:53,218 INFO [regionserver/cvp328:16201] wal.AbstractFSWAL: New WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646451113069 2022-03-05 03:31:56,400 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-0] handler.AssignRegionHandler: Open __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e. 2022-03-05 03:31:56,511 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-0] coprocessor.CoprocessorHost: System coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint loaded, priority=536870911. 2022-03-05 03:31:56,514 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-0] coprocessor.CoprocessorHost: System coprocessor org.apache.hadoop.hbase.coprocessor.AggregateImplementation loaded, priority=536870912. 2022-03-05 03:31:56,560 INFO [StoreOpener-4333b489291b1f77671b8f2e78779a1e-1] regionserver.HStore: Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family __status_family___ of region 4333b489291b1f77671b8f2e78779a1e 2022-03-05 03:31:56,618 INFO [StoreOpener-4333b489291b1f77671b8f2e78779a1e-1] compactions.CompactionConfiguration: size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4333b489291b1f77671b8f2e78779a1e columnFamilyName __status_family___ 2022-03-05 03:31:56,630 INFO [StoreOpener-4333b489291b1f77671b8f2e78779a1e-1] regionserver.HStore: Store=4333b489291b1f77671b8f2e78779a1e/__status_family___, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=FAST_DIFF, compression=NONE 2022-03-05 03:31:56,703 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-0] regionserver.HRegion: Opened 4333b489291b1f77671b8f2e78779a1e; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=109037380800, jitterRate=0.015489742159843445}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2022-03-05 03:31:56,737 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-0] regionserver.HRegionServer: Post open deploy tasks for __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., pid=13, masterSystemTime=1646451116212 2022-03-05 03:31:56,758 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-0] handler.AssignRegionHandler: Opened __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e. 2022-03-05 03:32:03,690 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter: Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2022-03-05 03:32:03,691 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter: Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.AggregateImplementation Metrics about HBase RegionObservers 2022-03-05 03:32:04,131 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 518 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 03:32:47,646 INFO [ReadOnlyZKClient-cvp328.sjc.aristanetworks.com:2181,cvp365.sjc.aristanetworks.com:2181,cvp90.sjc.aristanetworks.com:2181@0x7dafa7e3] zookeeper.ZooKeeper: Session: 0x10000006c9d01a4 closed 2022-03-05 03:32:47,646 INFO [ReadOnlyZKClient-cvp328.sjc.aristanetworks.com:2181,cvp365.sjc.aristanetworks.com:2181,cvp90.sjc.aristanetworks.com:2181@0x7dafa7e3-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x10000006c9d01a4 2022-03-05 03:34:46,737 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-1] handler.AssignRegionHandler: Open aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df. 2022-03-05 03:34:46,738 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-2] handler.AssignRegionHandler: Open aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb. 2022-03-05 03:34:46,738 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-0] handler.AssignRegionHandler: Open aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391. 2022-03-05 03:34:46,740 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-0] coprocessor.CoprocessorHost: System coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint loaded, priority=536870911. 2022-03-05 03:34:46,740 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-0] coprocessor.CoprocessorHost: System coprocessor org.apache.hadoop.hbase.coprocessor.AggregateImplementation loaded, priority=536870912. 2022-03-05 03:34:46,741 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-1] coprocessor.CoprocessorHost: System coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint loaded, priority=536870911. 2022-03-05 03:34:46,742 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-2] coprocessor.CoprocessorHost: System coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint loaded, priority=536870911. 2022-03-05 03:34:46,742 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-1] coprocessor.CoprocessorHost: System coprocessor org.apache.hadoop.hbase.coprocessor.AggregateImplementation loaded, priority=536870912. 2022-03-05 03:34:46,742 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-2] coprocessor.CoprocessorHost: System coprocessor org.apache.hadoop.hbase.coprocessor.AggregateImplementation loaded, priority=536870912. 2022-03-05 03:34:46,759 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-0] compress.CodecPool: Got brand-new compressor [.zst] 2022-03-05 03:34:46,759 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-2] compress.CodecPool: Got brand-new compressor [.zst] 2022-03-05 03:34:46,759 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-1] compress.CodecPool: Got brand-new compressor [.zst] 2022-03-05 03:34:46,778 INFO [StoreOpener-0a764e51258807790269e46c55b99391-1] regionserver.HStore: Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family e of region 0a764e51258807790269e46c55b99391 2022-03-05 03:34:46,779 INFO [StoreOpener-80a1ffe1dc2d8f258a6367d204dc64df-1] regionserver.HStore: Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family e of region 80a1ffe1dc2d8f258a6367d204dc64df 2022-03-05 03:34:46,780 INFO [StoreOpener-9b121c22e18472616debd0996ee2a4cb-1] regionserver.HStore: Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family e of region 9b121c22e18472616debd0996ee2a4cb 2022-03-05 03:34:46,794 INFO [StoreOpener-0a764e51258807790269e46c55b99391-1] compactions.CompactionConfiguration: size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0a764e51258807790269e46c55b99391 columnFamilyName e 2022-03-05 03:34:46,796 INFO [StoreOpener-80a1ffe1dc2d8f258a6367d204dc64df-1] compactions.CompactionConfiguration: size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 80a1ffe1dc2d8f258a6367d204dc64df columnFamilyName e 2022-03-05 03:34:46,798 INFO [StoreOpener-9b121c22e18472616debd0996ee2a4cb-1] compactions.CompactionConfiguration: size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9b121c22e18472616debd0996ee2a4cb columnFamilyName e 2022-03-05 03:34:46,802 INFO [StoreOpener-0a764e51258807790269e46c55b99391-1] regionserver.HStore: Store=0a764e51258807790269e46c55b99391/e, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=FAST_DIFF, compression=ZSTD 2022-03-05 03:34:46,802 INFO [StoreOpener-80a1ffe1dc2d8f258a6367d204dc64df-1] regionserver.HStore: Store=80a1ffe1dc2d8f258a6367d204dc64df/e, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=FAST_DIFF, compression=ZSTD 2022-03-05 03:34:46,803 INFO [StoreOpener-9b121c22e18472616debd0996ee2a4cb-1] regionserver.HStore: Store=9b121c22e18472616debd0996ee2a4cb/e, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=FAST_DIFF, compression=ZSTD 2022-03-05 03:34:46,841 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-2] regionserver.HRegion: Opened 9b121c22e18472616debd0996ee2a4cb; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=96857918400, jitterRate=-0.0979403406381607}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2022-03-05 03:34:46,841 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-1] regionserver.HRegion: Opened 80a1ffe1dc2d8f258a6367d204dc64df; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=120109760000, jitterRate=0.11860930919647217}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2022-03-05 03:34:46,844 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-0] regionserver.HRegion: Opened 0a764e51258807790269e46c55b99391; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=95032049600, jitterRate=-0.11494506895542145}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2022-03-05 03:34:46,847 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-2] regionserver.HRegionServer: Post open deploy tasks for aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., pid=55, masterSystemTime=1646451286727 2022-03-05 03:34:46,847 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-1] regionserver.HRegionServer: Post open deploy tasks for aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., pid=57, masterSystemTime=1646451286727 2022-03-05 03:34:46,847 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-0] regionserver.HRegionServer: Post open deploy tasks for aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391., pid=58, masterSystemTime=1646451286727 2022-03-05 03:34:46,858 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-1] handler.AssignRegionHandler: Opened aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df. 2022-03-05 03:34:46,858 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-1] handler.AssignRegionHandler: Open aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5. 2022-03-05 03:34:46,858 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-1] coprocessor.CoprocessorHost: System coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint loaded, priority=536870911. 2022-03-05 03:34:46,858 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-1] coprocessor.CoprocessorHost: System coprocessor org.apache.hadoop.hbase.coprocessor.AggregateImplementation loaded, priority=536870912. 2022-03-05 03:34:46,862 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-2] handler.AssignRegionHandler: Opened aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb. 2022-03-05 03:34:46,862 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-2] handler.AssignRegionHandler: Open aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb. 2022-03-05 03:34:46,863 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-2] coprocessor.CoprocessorHost: System coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint loaded, priority=536870911. 2022-03-05 03:34:46,863 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-2] coprocessor.CoprocessorHost: System coprocessor org.apache.hadoop.hbase.coprocessor.AggregateImplementation loaded, priority=536870912. 2022-03-05 03:34:46,866 INFO [StoreOpener-7534b3b1812808d2d3c16342fbf347a5-1] regionserver.HStore: Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family e of region 7534b3b1812808d2d3c16342fbf347a5 2022-03-05 03:34:46,867 INFO [StoreOpener-ccdacf4b414cb8d1bc548efc77c663cb-1] regionserver.HStore: Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family e of region ccdacf4b414cb8d1bc548efc77c663cb 2022-03-05 03:34:46,874 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-0] handler.AssignRegionHandler: Opened aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391. 2022-03-05 03:34:46,874 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-0] handler.AssignRegionHandler: Open aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. 2022-03-05 03:34:46,874 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-0] coprocessor.CoprocessorHost: System coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint loaded, priority=536870911. 2022-03-05 03:34:46,874 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-0] coprocessor.CoprocessorHost: System coprocessor org.apache.hadoop.hbase.coprocessor.AggregateImplementation loaded, priority=536870912. 2022-03-05 03:34:46,885 INFO [StoreOpener-7d1b53259f1d8280a47f7704f98f2c9f-1] regionserver.HStore: Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family e of region 7d1b53259f1d8280a47f7704f98f2c9f 2022-03-05 03:34:46,885 INFO [StoreOpener-7534b3b1812808d2d3c16342fbf347a5-1] compactions.CompactionConfiguration: size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7534b3b1812808d2d3c16342fbf347a5 columnFamilyName e 2022-03-05 03:34:46,887 INFO [StoreOpener-7534b3b1812808d2d3c16342fbf347a5-1] regionserver.HStore: Store=7534b3b1812808d2d3c16342fbf347a5/e, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=FAST_DIFF, compression=ZSTD 2022-03-05 03:34:46,896 INFO [StoreOpener-ccdacf4b414cb8d1bc548efc77c663cb-1] compactions.CompactionConfiguration: size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ccdacf4b414cb8d1bc548efc77c663cb columnFamilyName e 2022-03-05 03:34:46,897 INFO [StoreOpener-ccdacf4b414cb8d1bc548efc77c663cb-1] regionserver.HStore: Store=ccdacf4b414cb8d1bc548efc77c663cb/e, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=FAST_DIFF, compression=ZSTD 2022-03-05 03:34:46,908 INFO [StoreOpener-7d1b53259f1d8280a47f7704f98f2c9f-1] compactions.CompactionConfiguration: size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7d1b53259f1d8280a47f7704f98f2c9f columnFamilyName e 2022-03-05 03:34:46,914 INFO [StoreOpener-7d1b53259f1d8280a47f7704f98f2c9f-1] regionserver.HStore: Store=7d1b53259f1d8280a47f7704f98f2c9f/e, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=FAST_DIFF, compression=ZSTD 2022-03-05 03:34:46,937 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-1] regionserver.HRegion: Opened 7534b3b1812808d2d3c16342fbf347a5; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=102304304000, jitterRate=-0.04721692204475403}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2022-03-05 03:34:46,939 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-1] regionserver.HRegionServer: Post open deploy tasks for aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., pid=51, masterSystemTime=1646451286727 2022-03-05 03:34:46,946 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-1] handler.AssignRegionHandler: Opened aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5. 2022-03-05 03:34:46,950 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-2] regionserver.HRegion: Opened ccdacf4b414cb8d1bc548efc77c663cb; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=115290460800, jitterRate=0.07372608780860901}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2022-03-05 03:34:46,953 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-2] regionserver.HRegionServer: Post open deploy tasks for aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., pid=54, masterSystemTime=1646451286727 2022-03-05 03:34:46,953 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-0] regionserver.HRegion: Opened 7d1b53259f1d8280a47f7704f98f2c9f; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=116447352000, jitterRate=0.08450047671794891}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2022-03-05 03:34:46,954 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-0] regionserver.HRegionServer: Post open deploy tasks for aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., pid=56, masterSystemTime=1646451286727 2022-03-05 03:34:46,956 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-2] handler.AssignRegionHandler: Opened aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb. 2022-03-05 03:34:46,960 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-0] handler.AssignRegionHandler: Opened aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. 2022-03-05 03:34:55,725 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-0] handler.UnassignRegionHandler: Close 80a1ffe1dc2d8f258a6367d204dc64df 2022-03-05 03:34:55,725 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-1] handler.UnassignRegionHandler: Close 7534b3b1812808d2d3c16342fbf347a5 2022-03-05 03:34:55,726 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-1] regionserver.HRegion: Closing region aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5. 2022-03-05 03:34:55,726 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-0] regionserver.HRegion: Closing region aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df. 2022-03-05 03:34:55,726 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-2] handler.UnassignRegionHandler: Close 7d1b53259f1d8280a47f7704f98f2c9f 2022-03-05 03:34:55,727 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-2] regionserver.HRegion: Closing region aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. 2022-03-05 03:34:55,728 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-2] regionserver.HRegion: Flushing 7d1b53259f1d8280a47f7704f98f2c9f 1/1 column families, dataSize=221 B heapSize=784 B 2022-03-05 03:34:55,728 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-0] regionserver.HRegion: Flushing 80a1ffe1dc2d8f258a6367d204dc64df 1/1 column families, dataSize=386 B heapSize=1.05 KB 2022-03-05 03:34:55,744 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-1] regionserver.HRegion: Closed aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5. 2022-03-05 03:34:55,744 WARN [RS_CLOSE_REGION-regionserver/cvp328:16201-1] regionserver.HRegionServer: Not adding moved region record: 7534b3b1812808d2d3c16342fbf347a5 to self. 2022-03-05 03:34:55,749 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-1] handler.UnassignRegionHandler: Closed 7534b3b1812808d2d3c16342fbf347a5 2022-03-05 03:34:55,749 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-1] handler.UnassignRegionHandler: Close 9b121c22e18472616debd0996ee2a4cb 2022-03-05 03:34:55,749 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-1] regionserver.HRegion: Closing region aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb. 2022-03-05 03:34:55,749 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-1] regionserver.HRegion: Flushing 9b121c22e18472616debd0996ee2a4cb 1/1 column families, dataSize=672 B heapSize=1.57 KB 2022-03-05 03:34:55,770 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-1] compress.CodecPool: Got brand-new compressor [.zst] 2022-03-05 03:34:55,770 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-0] compress.CodecPool: Got brand-new compressor [.zst] 2022-03-05 03:34:55,770 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-2] compress.CodecPool: Got brand-new compressor [.zst] 2022-03-05 03:34:55,873 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-1] regionserver.DefaultStoreFlusher: Flushed memstore data size=672 B at sequenceid=13 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/.tmp/e/1b6efc90563e4d3c8fe0abed8ad2f6d2 2022-03-05 03:34:55,879 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-2] regionserver.DefaultStoreFlusher: Flushed memstore data size=221 B at sequenceid=7 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/.tmp/e/15f52eafa093478095f825d7f5718dd1 2022-03-05 03:34:55,879 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-0] regionserver.DefaultStoreFlusher: Flushed memstore data size=386 B at sequenceid=10 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/.tmp/e/ed2fa91de52e452292e744db7e9cea0f 2022-03-05 03:34:55,900 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-2] compress.CodecPool: Got brand-new decompressor [.zst] 2022-03-05 03:34:55,900 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-0] compress.CodecPool: Got brand-new decompressor [.zst] 2022-03-05 03:34:55,900 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-1] compress.CodecPool: Got brand-new decompressor [.zst] 2022-03-05 03:34:55,936 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-2] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/15f52eafa093478095f825d7f5718dd1, entries=3, sequenceid=7, filesize=4.9 K 2022-03-05 03:34:55,937 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/ed2fa91de52e452292e744db7e9cea0f, entries=4, sequenceid=10, filesize=4.8 K 2022-03-05 03:34:55,938 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/1b6efc90563e4d3c8fe0abed8ad2f6d2, entries=9, sequenceid=13, filesize=5.0 K 2022-03-05 03:34:55,940 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-2] regionserver.HRegion: Finished flush of dataSize ~221 B/221, heapSize ~768 B/768, currentSize=0 B/0 for 7d1b53259f1d8280a47f7704f98f2c9f in 212ms, sequenceid=7, compaction requested=false 2022-03-05 03:34:55,940 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-1] regionserver.HRegion: Finished flush of dataSize ~672 B/672, heapSize ~1.55 KB/1592, currentSize=0 B/0 for 9b121c22e18472616debd0996ee2a4cb in 191ms, sequenceid=13, compaction requested=false 2022-03-05 03:34:55,940 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-0] regionserver.HRegion: Finished flush of dataSize ~386 B/386, heapSize ~1.04 KB/1064, currentSize=0 B/0 for 80a1ffe1dc2d8f258a6367d204dc64df in 212ms, sequenceid=10, compaction requested=false 2022-03-05 03:34:55,963 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-2] regionserver.HRegion: Closed aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. 2022-03-05 03:34:55,963 WARN [RS_CLOSE_REGION-regionserver/cvp328:16201-2] regionserver.HRegionServer: Not adding moved region record: 7d1b53259f1d8280a47f7704f98f2c9f to self. 2022-03-05 03:34:55,965 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-1] regionserver.HRegion: Closed aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb. 2022-03-05 03:34:55,965 WARN [RS_CLOSE_REGION-regionserver/cvp328:16201-1] regionserver.HRegionServer: Not adding moved region record: 9b121c22e18472616debd0996ee2a4cb to self. 2022-03-05 03:34:55,968 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-2] handler.UnassignRegionHandler: Closed 7d1b53259f1d8280a47f7704f98f2c9f 2022-03-05 03:34:55,968 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-2] handler.UnassignRegionHandler: Close 0a764e51258807790269e46c55b99391 2022-03-05 03:34:55,968 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-0] regionserver.HRegion: Closed aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df. 2022-03-05 03:34:55,968 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-2] regionserver.HRegion: Closing region aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391. 2022-03-05 03:34:55,968 WARN [RS_CLOSE_REGION-regionserver/cvp328:16201-0] regionserver.HRegionServer: Not adding moved region record: 80a1ffe1dc2d8f258a6367d204dc64df to self. 2022-03-05 03:34:55,968 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-2] regionserver.HRegion: Flushing 0a764e51258807790269e46c55b99391 1/1 column families, dataSize=68 B heapSize=400 B 2022-03-05 03:34:55,969 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-1] handler.UnassignRegionHandler: Closed 9b121c22e18472616debd0996ee2a4cb 2022-03-05 03:34:55,969 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-1] handler.UnassignRegionHandler: Close ccdacf4b414cb8d1bc548efc77c663cb 2022-03-05 03:34:55,970 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-1] regionserver.HRegion: Closing region aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb. 2022-03-05 03:34:55,970 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-1] regionserver.HRegion: Flushing ccdacf4b414cb8d1bc548efc77c663cb 1/1 column families, dataSize=51 B heapSize=384 B 2022-03-05 03:34:55,976 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-0] handler.UnassignRegionHandler: Closed 80a1ffe1dc2d8f258a6367d204dc64df 2022-03-05 03:34:56,033 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-2] regionserver.DefaultStoreFlusher: Flushed memstore data size=68 B at sequenceid=5 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/.tmp/e/fee1de4b85094bf98768706dbe58df4c 2022-03-05 03:34:56,046 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-1] regionserver.DefaultStoreFlusher: Flushed memstore data size=51 B at sequenceid=5 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/.tmp/e/9c9dfd51990f4f16bd55aa8267b030fe 2022-03-05 03:34:56,082 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-1] handler.AssignRegionHandler: Open aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5. 2022-03-05 03:34:56,083 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-1] coprocessor.CoprocessorHost: System coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint loaded, priority=536870911. 2022-03-05 03:34:56,083 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-1] coprocessor.CoprocessorHost: System coprocessor org.apache.hadoop.hbase.coprocessor.AggregateImplementation loaded, priority=536870912. 2022-03-05 03:34:56,087 INFO [StoreOpener-7534b3b1812808d2d3c16342fbf347a5-1] regionserver.HStore: Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family e of region 7534b3b1812808d2d3c16342fbf347a5 2022-03-05 03:34:56,099 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-2] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/fee1de4b85094bf98768706dbe58df4c, entries=1, sequenceid=5, filesize=4.8 K 2022-03-05 03:34:56,102 INFO [StoreOpener-7534b3b1812808d2d3c16342fbf347a5-1] compactions.CompactionConfiguration: size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7534b3b1812808d2d3c16342fbf347a5 columnFamilyName e 2022-03-05 03:34:56,103 INFO [StoreOpener-7534b3b1812808d2d3c16342fbf347a5-1] regionserver.HStore: Store=7534b3b1812808d2d3c16342fbf347a5/e, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=FAST_DIFF, compression=ZSTD 2022-03-05 03:34:56,106 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-2] regionserver.HRegion: Finished flush of dataSize ~68 B/68, heapSize ~384 B/384, currentSize=0 B/0 for 0a764e51258807790269e46c55b99391 in 138ms, sequenceid=5, compaction requested=false 2022-03-05 03:34:56,116 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/9c9dfd51990f4f16bd55aa8267b030fe, entries=1, sequenceid=5, filesize=4.8 K 2022-03-05 03:34:56,124 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-1] regionserver.HRegion: Finished flush of dataSize ~51 B/51, heapSize ~368 B/368, currentSize=0 B/0 for ccdacf4b414cb8d1bc548efc77c663cb in 154ms, sequenceid=5, compaction requested=false 2022-03-05 03:34:56,125 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-1] regionserver.HRegion: Opened 7534b3b1812808d2d3c16342fbf347a5; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=110513876800, jitterRate=0.029240682721138}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2022-03-05 03:34:56,131 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-1] regionserver.HRegionServer: Post open deploy tasks for aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., pid=101, masterSystemTime=1646451296076 2022-03-05 03:34:56,131 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-2] regionserver.HRegion: Closed aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391. 2022-03-05 03:34:56,131 WARN [RS_CLOSE_REGION-regionserver/cvp328:16201-2] regionserver.HRegionServer: Not adding moved region record: 0a764e51258807790269e46c55b99391 to self. 2022-03-05 03:34:56,140 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-1] handler.AssignRegionHandler: Opened aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5. 2022-03-05 03:34:56,145 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-2] handler.UnassignRegionHandler: Closed 0a764e51258807790269e46c55b99391 2022-03-05 03:34:56,146 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-1] regionserver.HRegion: Closed aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb. 2022-03-05 03:34:56,146 WARN [RS_CLOSE_REGION-regionserver/cvp328:16201-1] regionserver.HRegionServer: Not adding moved region record: ccdacf4b414cb8d1bc548efc77c663cb to self. 2022-03-05 03:34:56,153 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-1] handler.UnassignRegionHandler: Closed ccdacf4b414cb8d1bc548efc77c663cb 2022-03-05 03:34:56,336 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-2] handler.AssignRegionHandler: Open aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb. 2022-03-05 03:34:56,337 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-2] coprocessor.CoprocessorHost: System coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint loaded, priority=536870911. 2022-03-05 03:34:56,337 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-2] coprocessor.CoprocessorHost: System coprocessor org.apache.hadoop.hbase.coprocessor.AggregateImplementation loaded, priority=536870912. 2022-03-05 03:34:56,337 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-0] handler.AssignRegionHandler: Open aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. 2022-03-05 03:34:56,337 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-0] coprocessor.CoprocessorHost: System coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint loaded, priority=536870911. 2022-03-05 03:34:56,338 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-0] coprocessor.CoprocessorHost: System coprocessor org.apache.hadoop.hbase.coprocessor.AggregateImplementation loaded, priority=536870912. 2022-03-05 03:34:56,339 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-1] handler.AssignRegionHandler: Open aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df. 2022-03-05 03:34:56,340 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-1] coprocessor.CoprocessorHost: System coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint loaded, priority=536870911. 2022-03-05 03:34:56,340 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-1] coprocessor.CoprocessorHost: System coprocessor org.apache.hadoop.hbase.coprocessor.AggregateImplementation loaded, priority=536870912. 2022-03-05 03:34:56,346 INFO [StoreOpener-7d1b53259f1d8280a47f7704f98f2c9f-1] regionserver.HStore: Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family e of region 7d1b53259f1d8280a47f7704f98f2c9f 2022-03-05 03:34:56,346 INFO [StoreOpener-9b121c22e18472616debd0996ee2a4cb-1] regionserver.HStore: Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family e of region 9b121c22e18472616debd0996ee2a4cb 2022-03-05 03:34:56,347 INFO [StoreOpener-80a1ffe1dc2d8f258a6367d204dc64df-1] regionserver.HStore: Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family e of region 80a1ffe1dc2d8f258a6367d204dc64df 2022-03-05 03:34:56,354 INFO [StoreOpener-7d1b53259f1d8280a47f7704f98f2c9f-1] compactions.CompactionConfiguration: size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7d1b53259f1d8280a47f7704f98f2c9f columnFamilyName e 2022-03-05 03:34:56,362 INFO [StoreOpener-9b121c22e18472616debd0996ee2a4cb-1] compactions.CompactionConfiguration: size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9b121c22e18472616debd0996ee2a4cb columnFamilyName e 2022-03-05 03:34:56,362 INFO [StoreOpener-80a1ffe1dc2d8f258a6367d204dc64df-1] compactions.CompactionConfiguration: size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 80a1ffe1dc2d8f258a6367d204dc64df columnFamilyName e 2022-03-05 03:34:56,398 INFO [StoreOpener-7d1b53259f1d8280a47f7704f98f2c9f-1] regionserver.HStore: Store=7d1b53259f1d8280a47f7704f98f2c9f/e, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=FAST_DIFF, compression=ZSTD 2022-03-05 03:34:56,405 INFO [StoreOpener-80a1ffe1dc2d8f258a6367d204dc64df-1] regionserver.HStore: Store=80a1ffe1dc2d8f258a6367d204dc64df/e, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=FAST_DIFF, compression=ZSTD 2022-03-05 03:34:56,409 INFO [StoreOpener-9b121c22e18472616debd0996ee2a4cb-1] regionserver.HStore: Store=9b121c22e18472616debd0996ee2a4cb/e, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=FAST_DIFF, compression=ZSTD 2022-03-05 03:34:56,440 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-0] regionserver.HRegion: Opened 7d1b53259f1d8280a47f7704f98f2c9f; next sequenceid=11; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=103294417600, jitterRate=-0.03799577057361603}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2022-03-05 03:34:56,440 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-1] regionserver.HRegion: Opened 80a1ffe1dc2d8f258a6367d204dc64df; next sequenceid=14; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=107238563200, jitterRate=-0.001263052225112915}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2022-03-05 03:34:56,441 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-2] regionserver.HRegion: Opened 9b121c22e18472616debd0996ee2a4cb; next sequenceid=17; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=117040401600, jitterRate=0.09002368152141571}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2022-03-05 03:34:56,443 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-0] regionserver.HRegionServer: Post open deploy tasks for aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., pid=109, masterSystemTime=1646451296328 2022-03-05 03:34:56,443 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-1] regionserver.HRegionServer: Post open deploy tasks for aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., pid=111, masterSystemTime=1646451296328 2022-03-05 03:34:56,443 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-2] regionserver.HRegionServer: Post open deploy tasks for aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., pid=110, masterSystemTime=1646451296328 2022-03-05 03:34:56,448 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-1] handler.AssignRegionHandler: Opened aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df. 2022-03-05 03:34:56,449 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-0] handler.AssignRegionHandler: Opened aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. 2022-03-05 03:34:56,450 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-2] handler.AssignRegionHandler: Opened aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb. 2022-03-05 03:34:56,535 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-1] handler.AssignRegionHandler: Open aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391. 2022-03-05 03:34:56,535 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-1] coprocessor.CoprocessorHost: System coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint loaded, priority=536870911. 2022-03-05 03:34:56,535 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-1] coprocessor.CoprocessorHost: System coprocessor org.apache.hadoop.hbase.coprocessor.AggregateImplementation loaded, priority=536870912. 2022-03-05 03:34:56,536 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-0] handler.AssignRegionHandler: Open aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb. 2022-03-05 03:34:56,536 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-0] coprocessor.CoprocessorHost: System coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint loaded, priority=536870911. 2022-03-05 03:34:56,536 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-0] coprocessor.CoprocessorHost: System coprocessor org.apache.hadoop.hbase.coprocessor.AggregateImplementation loaded, priority=536870912. 2022-03-05 03:34:56,538 INFO [StoreOpener-0a764e51258807790269e46c55b99391-1] regionserver.HStore: Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family e of region 0a764e51258807790269e46c55b99391 2022-03-05 03:34:56,539 INFO [StoreOpener-ccdacf4b414cb8d1bc548efc77c663cb-1] regionserver.HStore: Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family e of region ccdacf4b414cb8d1bc548efc77c663cb 2022-03-05 03:34:56,544 INFO [StoreOpener-0a764e51258807790269e46c55b99391-1] compactions.CompactionConfiguration: size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0a764e51258807790269e46c55b99391 columnFamilyName e 2022-03-05 03:34:56,544 INFO [StoreOpener-ccdacf4b414cb8d1bc548efc77c663cb-1] compactions.CompactionConfiguration: size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ccdacf4b414cb8d1bc548efc77c663cb columnFamilyName e 2022-03-05 03:34:56,560 INFO [StoreOpener-0a764e51258807790269e46c55b99391-1] regionserver.HStore: Store=0a764e51258807790269e46c55b99391/e, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=FAST_DIFF, compression=ZSTD 2022-03-05 03:34:56,563 INFO [StoreOpener-ccdacf4b414cb8d1bc548efc77c663cb-1] regionserver.HStore: Store=ccdacf4b414cb8d1bc548efc77c663cb/e, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=FAST_DIFF, compression=ZSTD 2022-03-05 03:34:56,576 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-1] regionserver.HRegion: Opened 0a764e51258807790269e46c55b99391; next sequenceid=9; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=98151817600, jitterRate=-0.08588996529579163}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2022-03-05 03:34:56,577 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-0] regionserver.HRegion: Opened ccdacf4b414cb8d1bc548efc77c663cb; next sequenceid=9; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=115465750400, jitterRate=0.07535859942436218}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2022-03-05 03:34:56,578 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-1] regionserver.HRegionServer: Post open deploy tasks for aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391., pid=116, masterSystemTime=1646451296527 2022-03-05 03:34:56,579 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-0] regionserver.HRegionServer: Post open deploy tasks for aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., pid=113, masterSystemTime=1646451296527 2022-03-05 03:34:56,592 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-1] handler.AssignRegionHandler: Opened aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391. 2022-03-05 03:34:56,593 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-0] handler.AssignRegionHandler: Opened aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb. 2022-03-05 03:35:06,335 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-0] handler.UnassignRegionHandler: Close ccdacf4b414cb8d1bc548efc77c663cb 2022-03-05 03:35:06,335 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-2] handler.UnassignRegionHandler: Close 7d1b53259f1d8280a47f7704f98f2c9f 2022-03-05 03:35:06,335 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-1] handler.UnassignRegionHandler: Close 80a1ffe1dc2d8f258a6367d204dc64df 2022-03-05 03:35:06,335 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-0] regionserver.HRegion: Closing region aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb. 2022-03-05 03:35:06,336 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-2] regionserver.HRegion: Closing region aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. 2022-03-05 03:35:06,336 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-1] regionserver.HRegion: Closing region aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df. 2022-03-05 03:35:06,358 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-2] regionserver.HRegion: Closed aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. 2022-03-05 03:35:06,358 WARN [RS_CLOSE_REGION-regionserver/cvp328:16201-2] regionserver.HRegionServer: Not adding moved region record: 7d1b53259f1d8280a47f7704f98f2c9f to self. 2022-03-05 03:35:06,363 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-2] handler.UnassignRegionHandler: Closed 7d1b53259f1d8280a47f7704f98f2c9f 2022-03-05 03:35:06,363 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-2] handler.UnassignRegionHandler: Close 9b121c22e18472616debd0996ee2a4cb 2022-03-05 03:35:06,364 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-2] regionserver.HRegion: Closing region aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb. 2022-03-05 03:35:06,366 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-0] regionserver.HRegion: Closed aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb. 2022-03-05 03:35:06,366 WARN [RS_CLOSE_REGION-regionserver/cvp328:16201-0] regionserver.HRegionServer: Not adding moved region record: ccdacf4b414cb8d1bc548efc77c663cb to self. 2022-03-05 03:35:06,367 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-1] regionserver.HRegion: Closed aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df. 2022-03-05 03:35:06,367 WARN [RS_CLOSE_REGION-regionserver/cvp328:16201-1] regionserver.HRegionServer: Not adding moved region record: 80a1ffe1dc2d8f258a6367d204dc64df to self. 2022-03-05 03:35:06,381 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-0] handler.UnassignRegionHandler: Closed ccdacf4b414cb8d1bc548efc77c663cb 2022-03-05 03:35:06,382 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-0] handler.UnassignRegionHandler: Close 0a764e51258807790269e46c55b99391 2022-03-05 03:35:06,382 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-0] regionserver.HRegion: Closing region aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391. 2022-03-05 03:35:06,389 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-1] handler.UnassignRegionHandler: Closed 80a1ffe1dc2d8f258a6367d204dc64df 2022-03-05 03:35:06,389 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-1] handler.UnassignRegionHandler: Close 7534b3b1812808d2d3c16342fbf347a5 2022-03-05 03:35:06,393 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-1] regionserver.HRegion: Closing region aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5. 2022-03-05 03:35:06,417 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-2] regionserver.HRegion: Closed aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb. 2022-03-05 03:35:06,417 WARN [RS_CLOSE_REGION-regionserver/cvp328:16201-2] regionserver.HRegionServer: Not adding moved region record: 9b121c22e18472616debd0996ee2a4cb to self. 2022-03-05 03:35:06,431 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-2] handler.UnassignRegionHandler: Closed 9b121c22e18472616debd0996ee2a4cb 2022-03-05 03:35:06,444 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-1] regionserver.HRegion: Closed aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5. 2022-03-05 03:35:06,444 WARN [RS_CLOSE_REGION-regionserver/cvp328:16201-1] regionserver.HRegionServer: Not adding moved region record: 7534b3b1812808d2d3c16342fbf347a5 to self. 2022-03-05 03:35:06,446 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-0] regionserver.HRegion: Closed aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391. 2022-03-05 03:35:06,446 WARN [RS_CLOSE_REGION-regionserver/cvp328:16201-0] regionserver.HRegionServer: Not adding moved region record: 0a764e51258807790269e46c55b99391 to self. 2022-03-05 03:35:06,448 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-1] handler.UnassignRegionHandler: Closed 7534b3b1812808d2d3c16342fbf347a5 2022-03-05 03:35:06,449 INFO [RS_CLOSE_REGION-regionserver/cvp328:16201-0] handler.UnassignRegionHandler: Closed 0a764e51258807790269e46c55b99391 2022-03-05 03:35:06,719 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-2] handler.AssignRegionHandler: Open aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5. 2022-03-05 03:35:06,719 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-1] handler.AssignRegionHandler: Open aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. 2022-03-05 03:35:06,719 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-0] handler.AssignRegionHandler: Open aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391. 2022-03-05 03:35:06,720 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-1] coprocessor.CoprocessorHost: System coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint loaded, priority=536870911. 2022-03-05 03:35:06,720 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-2] coprocessor.CoprocessorHost: System coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint loaded, priority=536870911. 2022-03-05 03:35:06,720 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-1] coprocessor.CoprocessorHost: System coprocessor org.apache.hadoop.hbase.coprocessor.AggregateImplementation loaded, priority=536870912. 2022-03-05 03:35:06,720 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-2] coprocessor.CoprocessorHost: System coprocessor org.apache.hadoop.hbase.coprocessor.AggregateImplementation loaded, priority=536870912. 2022-03-05 03:35:06,720 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-0] coprocessor.CoprocessorHost: System coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint loaded, priority=536870911. 2022-03-05 03:35:06,720 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-0] coprocessor.CoprocessorHost: System coprocessor org.apache.hadoop.hbase.coprocessor.AggregateImplementation loaded, priority=536870912. 2022-03-05 03:35:06,722 INFO [StoreOpener-7d1b53259f1d8280a47f7704f98f2c9f-1] regionserver.HStore: Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family e of region 7d1b53259f1d8280a47f7704f98f2c9f 2022-03-05 03:35:06,723 INFO [StoreOpener-7534b3b1812808d2d3c16342fbf347a5-1] regionserver.HStore: Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family e of region 7534b3b1812808d2d3c16342fbf347a5 2022-03-05 03:35:06,723 INFO [StoreOpener-0a764e51258807790269e46c55b99391-1] regionserver.HStore: Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family e of region 0a764e51258807790269e46c55b99391 2022-03-05 03:35:06,729 INFO [StoreOpener-7d1b53259f1d8280a47f7704f98f2c9f-1] compactions.CompactionConfiguration: size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7d1b53259f1d8280a47f7704f98f2c9f columnFamilyName e 2022-03-05 03:35:06,734 INFO [StoreOpener-7534b3b1812808d2d3c16342fbf347a5-1] compactions.CompactionConfiguration: size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7534b3b1812808d2d3c16342fbf347a5 columnFamilyName e 2022-03-05 03:35:06,735 INFO [StoreOpener-0a764e51258807790269e46c55b99391-1] compactions.CompactionConfiguration: size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0a764e51258807790269e46c55b99391 columnFamilyName e 2022-03-05 03:35:06,737 INFO [StoreOpener-7534b3b1812808d2d3c16342fbf347a5-1] regionserver.HStore: Store=7534b3b1812808d2d3c16342fbf347a5/e, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=FAST_DIFF, compression=ZSTD 2022-03-05 03:35:06,750 INFO [StoreOpener-0a764e51258807790269e46c55b99391-1] regionserver.HStore: Store=0a764e51258807790269e46c55b99391/e, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=FAST_DIFF, compression=ZSTD 2022-03-05 03:35:06,752 INFO [StoreOpener-7d1b53259f1d8280a47f7704f98f2c9f-1] regionserver.HStore: Store=7d1b53259f1d8280a47f7704f98f2c9f/e, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=FAST_DIFF, compression=ZSTD 2022-03-05 03:35:06,763 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-2] regionserver.HRegion: Opened 7534b3b1812808d2d3c16342fbf347a5; next sequenceid=8; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=118710956800, jitterRate=0.10558193922042847}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2022-03-05 03:35:06,765 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-2] regionserver.HRegionServer: Post open deploy tasks for aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., pid=177, masterSystemTime=1646451306713 2022-03-05 03:35:06,770 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-2] handler.AssignRegionHandler: Opened aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5. 2022-03-05 03:35:06,770 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-2] handler.AssignRegionHandler: Open aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb. 2022-03-05 03:35:06,771 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-2] coprocessor.CoprocessorHost: System coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint loaded, priority=536870911. 2022-03-05 03:35:06,771 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-2] coprocessor.CoprocessorHost: System coprocessor org.apache.hadoop.hbase.coprocessor.AggregateImplementation loaded, priority=536870912. 2022-03-05 03:35:06,777 INFO [StoreOpener-ccdacf4b414cb8d1bc548efc77c663cb-1] regionserver.HStore: Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family e of region ccdacf4b414cb8d1bc548efc77c663cb 2022-03-05 03:35:06,779 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-1] regionserver.HRegion: Opened 7d1b53259f1d8280a47f7704f98f2c9f; next sequenceid=14; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=109758873600, jitterRate=0.02220916748046875}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2022-03-05 03:35:06,779 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-0] regionserver.HRegion: Opened 0a764e51258807790269e46c55b99391; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=94154230400, jitterRate=-0.12312039732933044}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2022-03-05 03:35:06,782 INFO [StoreOpener-ccdacf4b414cb8d1bc548efc77c663cb-1] compactions.CompactionConfiguration: size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ccdacf4b414cb8d1bc548efc77c663cb columnFamilyName e 2022-03-05 03:35:06,783 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-1] regionserver.HRegionServer: Post open deploy tasks for aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., pid=179, masterSystemTime=1646451306713 2022-03-05 03:35:06,783 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-0] regionserver.HRegionServer: Post open deploy tasks for aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391., pid=182, masterSystemTime=1646451306713 2022-03-05 03:35:06,786 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-1] handler.AssignRegionHandler: Opened aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. 2022-03-05 03:35:06,787 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-1] handler.AssignRegionHandler: Open aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb. 2022-03-05 03:35:06,787 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-1] coprocessor.CoprocessorHost: System coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint loaded, priority=536870911. 2022-03-05 03:35:06,787 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-1] coprocessor.CoprocessorHost: System coprocessor org.apache.hadoop.hbase.coprocessor.AggregateImplementation loaded, priority=536870912. 2022-03-05 03:35:06,793 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-0] handler.AssignRegionHandler: Opened aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391. 2022-03-05 03:35:06,793 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-0] handler.AssignRegionHandler: Open aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df. 2022-03-05 03:35:06,793 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-0] coprocessor.CoprocessorHost: System coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint loaded, priority=536870911. 2022-03-05 03:35:06,793 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-0] coprocessor.CoprocessorHost: System coprocessor org.apache.hadoop.hbase.coprocessor.AggregateImplementation loaded, priority=536870912. 2022-03-05 03:35:06,797 INFO [StoreOpener-9b121c22e18472616debd0996ee2a4cb-1] regionserver.HStore: Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family e of region 9b121c22e18472616debd0996ee2a4cb 2022-03-05 03:35:06,803 INFO [StoreOpener-80a1ffe1dc2d8f258a6367d204dc64df-1] regionserver.HStore: Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family e of region 80a1ffe1dc2d8f258a6367d204dc64df 2022-03-05 03:35:06,806 INFO [StoreOpener-9b121c22e18472616debd0996ee2a4cb-1] compactions.CompactionConfiguration: size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9b121c22e18472616debd0996ee2a4cb columnFamilyName e 2022-03-05 03:35:06,806 INFO [StoreOpener-ccdacf4b414cb8d1bc548efc77c663cb-1] regionserver.HStore: Store=ccdacf4b414cb8d1bc548efc77c663cb/e, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=FAST_DIFF, compression=ZSTD 2022-03-05 03:35:06,810 INFO [StoreOpener-80a1ffe1dc2d8f258a6367d204dc64df-1] compactions.CompactionConfiguration: size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 80a1ffe1dc2d8f258a6367d204dc64df columnFamilyName e 2022-03-05 03:35:06,836 INFO [StoreOpener-9b121c22e18472616debd0996ee2a4cb-1] regionserver.HStore: Store=9b121c22e18472616debd0996ee2a4cb/e, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=FAST_DIFF, compression=ZSTD 2022-03-05 03:35:06,838 INFO [StoreOpener-80a1ffe1dc2d8f258a6367d204dc64df-1] regionserver.HStore: Store=80a1ffe1dc2d8f258a6367d204dc64df/e, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=FAST_DIFF, compression=ZSTD 2022-03-05 03:35:06,841 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-2] regionserver.HRegion: Opened ccdacf4b414cb8d1bc548efc77c663cb; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=115451040000, jitterRate=0.07522159814834595}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2022-03-05 03:35:06,844 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-2] regionserver.HRegionServer: Post open deploy tasks for aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., pid=178, masterSystemTime=1646451306713 2022-03-05 03:35:06,851 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-2] handler.AssignRegionHandler: Opened aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb. 2022-03-05 03:35:06,862 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-1] regionserver.HRegion: Opened 9b121c22e18472616debd0996ee2a4cb; next sequenceid=20; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=110936208000, jitterRate=0.033173948526382446}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2022-03-05 03:35:06,863 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-0] regionserver.HRegion: Opened 80a1ffe1dc2d8f258a6367d204dc64df; next sequenceid=17; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=119703784000, jitterRate=0.11482836306095123}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2022-03-05 03:35:06,863 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-1] regionserver.HRegionServer: Post open deploy tasks for aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., pid=180, masterSystemTime=1646451306713 2022-03-05 03:35:06,864 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-0] regionserver.HRegionServer: Post open deploy tasks for aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., pid=181, masterSystemTime=1646451306713 2022-03-05 03:35:06,866 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-1] handler.AssignRegionHandler: Opened aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb. 2022-03-05 03:35:06,869 INFO [RS_OPEN_REGION-regionserver/cvp328:16201-0] handler.AssignRegionHandler: Opened aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df. 2022-03-05 03:36:04,289 INFO [RS-EventLoopGroup-1-5] compress.CodecPool: Got brand-new decompressor [.snappy] 2022-03-05 03:36:04,297 INFO [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] compress.CodecPool: Got brand-new compressor [.snappy] 2022-03-05 03:36:04,368 INFO [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] compress.CodecPool: Got brand-new compressor [.snappy] 2022-03-05 03:36:04,368 INFO [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] compress.CodecPool: Got brand-new compressor [.snappy] 2022-03-05 03:36:04,370 INFO [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] compress.CodecPool: Got brand-new compressor [.snappy] 2022-03-05 03:36:19,723 INFO [RS-EventLoopGroup-1-1] compress.CodecPool: Got brand-new decompressor [.snappy] 2022-03-05 03:36:27,413 INFO [RS-EventLoopGroup-1-5] compress.CodecPool: Got brand-new decompressor [.snappy] 2022-03-05 03:36:44,383 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=237.11 KB, freeSize=306.97 MB, max=307.20 MB, blockCount=8, accesses=501, hits=493, hitRatio=98.40%, , cachingAccesses=501, cachingHits=493, cachingHitsRatio=98.40%, evictions=29, evicted=0, evictedPerRun=0.0 2022-03-05 03:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 03:36:54,988 INFO [RS_OPEN_META-regionserver/cvp328:16201-0] handler.AssignRegionHandler: Open hbase:meta,,1.1588230740 2022-03-05 03:36:54,988 INFO [RS_OPEN_META-regionserver/cvp328:16201-0] wal.WALFactory: Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2022-03-05 03:36:54,991 INFO [RS_OPEN_META-regionserver/cvp328:16201-0] wal.AbstractFSWAL: WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta, suffix=.meta, logDir=hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340, archiveDir=hdfs://mycluster/hbase/oldWALs, maxLogs=32 2022-03-05 03:36:55,005 INFO [RS_OPEN_META-regionserver/cvp328:16201-0] wal.AbstractFSWAL: New WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646451414992.meta 2022-03-05 03:36:55,008 INFO [RS_OPEN_META-regionserver/cvp328:16201-0] regionserver.RegionCoprocessorHost: Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2022-03-05 03:36:55,010 INFO [StoreOpener-1588230740-1] regionserver.HStore: Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2022-03-05 03:36:55,013 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration: size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2022-03-05 03:36:55,019 INFO [StoreOpener-1588230740-1] regionserver.HStore: Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2022-03-05 03:36:55,019 INFO [StoreOpener-1588230740-1] regionserver.HStore: Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2022-03-05 03:36:55,023 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration: size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2022-03-05 03:36:55,023 INFO [StoreOpener-1588230740-1] regionserver.HStore: Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2022-03-05 03:36:55,023 INFO [StoreOpener-1588230740-1] regionserver.HStore: Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2022-03-05 03:36:55,026 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration: size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2022-03-05 03:36:55,032 INFO [StoreOpener-1588230740-1] regionserver.HStore: Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2022-03-05 03:36:55,040 INFO [RS_OPEN_META-regionserver/cvp328:16201-0] regionserver.HRegion: Opened 1588230740; next sequenceid=231; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=107111027200, jitterRate=-0.0024508237838745117}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2022-03-05 03:36:55,044 INFO [RS_OPEN_META-regionserver/cvp328:16201-0] regionserver.HRegionServer: Post open deploy tasks for hbase:meta,,1.1588230740, pid=185, masterSystemTime=1646451414983 2022-03-05 03:36:55,048 INFO [RS_OPEN_META-regionserver/cvp328:16201-0] handler.AssignRegionHandler: Opened hbase:meta,,1.1588230740 2022-03-05 03:39:38,274 INFO [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] compress.CodecPool: Got brand-new compressor [.snappy] 2022-03-05 03:39:39,817 INFO [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] compress.CodecPool: Got brand-new compressor [.snappy] 2022-03-05 03:39:39,818 INFO [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] compress.CodecPool: Got brand-new compressor [.snappy] 2022-03-05 03:39:40,128 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] compress.CodecPool: Got brand-new compressor [.snappy] 2022-03-05 03:39:40,128 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] compress.CodecPool: Got brand-new compressor [.snappy] 2022-03-05 03:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=288.27 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=14, accesses=3575, hits=3561, hitRatio=99.61%, , cachingAccesses=3575, cachingHits=3561, cachingHitsRatio=99.61%, evictions=59, evicted=0, evictedPerRun=0.0 2022-03-05 03:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 03:41:58,060 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646451113069 with entries=5360, filesize=2.78 MB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646451718029 2022-03-05 03:43:09,291 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1396 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 03:43:09,291 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1040 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 03:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=289.55 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=15, accesses=3603, hits=3588, hitRatio=99.58%, , cachingAccesses=3603, cachingHits=3588, cachingHitsRatio=99.58%, evictions=89, evicted=0, evictedPerRun=0.0 2022-03-05 03:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 03:46:58,076 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646451414992.meta with entries=1, filesize=400 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646452018065.meta 2022-03-05 03:46:58,077 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646451414992.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646451414992.meta 2022-03-05 03:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=289.55 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=15, accesses=3625, hits=3610, hitRatio=99.59%, , cachingAccesses=3625, cachingHits=3610, cachingHitsRatio=99.59%, evictions=119, evicted=0, evictedPerRun=0.0 2022-03-05 03:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 03:51:58,097 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646451718029 with entries=1236, filesize=621.33 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646452318081 2022-03-05 03:53:06,927 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3326 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 03:53:06,927 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3322 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 03:53:06,928 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2757 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 03:53:06,928 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 621 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 03:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=289.55 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=15, accesses=3647, hits=3632, hitRatio=99.59%, , cachingAccesses=3647, cachingHits=3632, cachingHitsRatio=99.59%, evictions=149, evicted=0, evictedPerRun=0.0 2022-03-05 03:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 03:56:58,121 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646452018065.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646452618102.meta 2022-03-05 03:56:58,122 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646452018065.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646452018065.meta 2022-03-05 03:58:01,729 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 631 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:00:08,114 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1299ms No GCs detected 2022-03-05 04:00:08,797 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 678 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=289.55 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=15, accesses=3669, hits=3654, hitRatio=99.59%, , cachingAccesses=3669, cachingHits=3654, cachingHitsRatio=99.59%, evictions=179, evicted=0, evictedPerRun=0.0 2022-03-05 04:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 04:01:58,140 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646452318081 with entries=1259, filesize=557.05 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646452918125 2022-03-05 04:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=289.55 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=15, accesses=3697, hits=3682, hitRatio=99.59%, , cachingAccesses=3697, cachingHits=3682, cachingHitsRatio=99.59%, evictions=209, evicted=0, evictedPerRun=0.0 2022-03-05 04:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 04:06:58,183 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646452618102.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646453218144.meta 2022-03-05 04:06:58,183 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646452618102.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646452618102.meta 2022-03-05 04:07:31,313 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 583 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 04:07:31,313 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 583 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 04:07:43,133 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1021 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 04:07:58,940 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.sync(AsyncProtobufLogWriter.java:148) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:394) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.consume(AsyncFSWAL.java:581) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 04:07:59,652 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 5244ms No GCs detected 2022-03-05 04:08:02,918 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646453218144.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646453278917.meta 2022-03-05 04:08:02,919 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646453218144.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646453218144.meta 2022-03-05 04:08:02,921 WARN [Close-WAL-Writer-3] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 04:08:02,921 WARN [Close-WAL-Writer-3] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 04:08:02,924 INFO [Close-WAL-Writer-3] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646453218144.meta 2022-03-05 04:08:02,968 INFO [Close-WAL-Writer-3] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646453218144.meta after 44ms 2022-03-05 04:08:03,001 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646452918125 with entries=767, filesize=353.76 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646453282918 2022-03-05 04:08:03,001 WARN [Close-WAL-Writer-3] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 04:08:03,006 WARN [Close-WAL-Writer-3] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 04:08:03,008 INFO [Close-WAL-Writer-3] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646452918125 2022-03-05 04:08:03,046 INFO [Close-WAL-Writer-3] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646452918125 after 37ms 2022-03-05 04:08:06,979 WARN [Close-WAL-Writer-3] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646453218144.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646453218144.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 04:08:07,046 INFO [Close-WAL-Writer-3] util.RecoverLeaseFSUtils: Recovered lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646452918125 after 4037ms 2022-03-05 04:08:08,762 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 879 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 04:08:09,715 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1475 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 04:08:20,568 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1352ms No GCs detected 2022-03-05 04:08:20,660 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 10016 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 04:08:20,660 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Requesting log roll because we exceeded slow sync threshold; time=10016 ms, threshold=10000 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 04:08:20,737 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646453290643","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":10017,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 04:08:20,789 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 9473 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 04:08:20,789 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 8869 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 04:08:20,789 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 8468 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 04:08:24,025 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 598 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 04:08:24,040 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646453282918 with entries=65, filesize=30.25 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646453300668 2022-03-05 04:08:25,701 INFO [Close-WAL-Writer-3] retry.RetryInvocationHandler: at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:614) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.complete over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 04:08:25,708 INFO [Close-WAL-Writer-3] retry.RetryInvocationHandler: ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:614) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.complete over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 1 failover attempts. Trying to failover after sleeping for 411ms. 2022-03-05 04:08:26,121 INFO [Close-WAL-Writer-3] retry.RetryInvocationHandler: ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:614) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.complete over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 2 failover attempts. Trying to failover after sleeping for 841ms. 2022-03-05 04:08:27,435 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 982 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:08:28,846 INFO [Close-WAL-Writer-3] retry.RetryInvocationHandler: lientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:614) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.complete over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 3 failover attempts. Trying to failover after sleeping for 3420ms. 2022-03-05 04:08:28,851 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 938 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:08:28,851 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 937 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:08:28,851 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 937 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:08:32,268 INFO [Close-WAL-Writer-3] retry.RetryInvocationHandler: lientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:614) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.complete over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 4 failover attempts. Trying to failover after sleeping for 6984ms. 2022-03-05 04:08:32,835 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 04:08:33,464 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2164 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:08:33,464 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2063 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:08:33,464 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1334 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:08:33,464 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1334 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:08:33,464 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 732 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:08:33,464 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 732 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:08:36,026 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2131 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:08:36,028 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2129 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:08:36,028 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1655 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:08:37,911 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1744 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:08:38,762 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 850 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:08:39,252 WARN [Close-WAL-Writer-3] retry.RetryInvocationHandler: A failover has occurred since the start of call #780 ClientNamenodeProtocolTranslatorPB.complete over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 2022-03-05 04:08:41,067 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2854 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:08:41,067 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1733 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:08:41,473 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 776 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:08:46,876 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 912 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:08:48,551 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1283 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:08:50,608 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2056 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:08:50,608 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2054 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:08:53,457 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4890 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:08:53,457 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2844 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:08:55,744 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1831 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:08:55,744 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1112 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:09:03,979 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 699 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:09:12,648 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 510 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:09:13,426 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 517 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:10:54,129 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 889 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:10:54,129 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 800 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:10:57,253 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1256 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:10:58,960 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2760 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:10:58,960 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2011 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:11:01,128 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 765 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:11:01,824 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 848 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:11:01,824 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 759 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:11:01,824 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 698 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:11:01,824 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 697 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:11:16,483 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 04:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=289.55 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=15, accesses=3719, hits=3704, hitRatio=99.60%, , cachingAccesses=3719, cachingHits=3704, cachingHitsRatio=99.60%, evictions=239, evicted=0, evictedPerRun=0.0 2022-03-05 04:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 04:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=289.55 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=15, accesses=3741, hits=3726, hitRatio=99.60%, , cachingAccesses=3741, cachingHits=3726, cachingHitsRatio=99.60%, evictions=269, evicted=0, evictedPerRun=0.0 2022-03-05 04:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 04:16:53,691 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1307 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:17:16,023 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3906 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:17:16,341 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3722ms No GCs detected 2022-03-05 04:17:19,203 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2810 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:17:19,203 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2718 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:17:19,205 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1829ms No GCs detected 2022-03-05 04:17:30,960 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1153 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:17:30,960 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 724 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:17:30,960 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 578 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:17:38,278 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4188ms No GCs detected 2022-03-05 04:17:50,250 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 04:18:09,260 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646453278917.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646453888276.meta 2022-03-05 04:18:09,260 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646453278917.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646453278917.meta 2022-03-05 04:18:12,978 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1425ms No GCs detected 2022-03-05 04:18:25,802 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 7313ms No GCs detected 2022-03-05 04:18:25,844 INFO [regionserver/cvp328:16201.Chore.1] hbase.ScheduledChore: Chore: CompactionChecker missed its start time 2022-03-05 04:18:25,844 INFO [regionserver/cvp328:16201.Chore.1] hbase.ScheduledChore: Chore: MemstoreFlusherChore missed its start time 2022-03-05 04:18:27,455 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646453300668 with entries=1306, filesize=577.10 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646453905776 2022-03-05 04:18:37,474 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1825 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:18:39,549 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1367 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:18:47,579 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646453888276.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646453927462.meta 2022-03-05 04:18:47,580 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646453888276.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646453888276.meta 2022-03-05 04:18:47,582 WARN [Close-WAL-Writer-4] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 04:18:47,582 WARN [Close-WAL-Writer-4] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 04:18:47,582 INFO [Close-WAL-Writer-4] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646453888276.meta 2022-03-05 04:18:47,661 INFO [Close-WAL-Writer-4] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646453888276.meta after 79ms 2022-03-05 04:18:51,664 WARN [Close-WAL-Writer-4] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646453888276.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646453888276.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 04:21:38,442 WARN [regionserver/cvp328:16201] util.Sleeper: We slept 15771ms instead of 5000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired 2022-03-05 04:21:38,911 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4290ms No GCs detected 2022-03-05 04:21:38,911 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 15615 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:21:38,944 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Requesting log roll because we exceeded slow sync threshold; time=15615 ms, threshold=10000 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:21:38,944 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.sync(AsyncProtobufLogWriter.java:148) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:394) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.consume(AsyncFSWAL.java:581) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 04:21:38,968 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646454083296","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":15618,"client":"172.30.41.155:52176","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 04:21:40,256 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646453905776 with entries=405, filesize=182.15 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646454098957 2022-03-05 04:21:40,256 WARN [Close-WAL-Writer-5] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 04:21:40,257 WARN [Close-WAL-Writer-5] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 04:21:40,257 INFO [Close-WAL-Writer-5] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646453905776 2022-03-05 04:21:40,265 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646454088476","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":11788,"client":"172.30.41.155:52176","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 04:21:40,265 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646454088476","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":11789,"client":"172.30.41.155:52176","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 04:21:40,266 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646454088476","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":11789,"client":"172.30.41.155:52176","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-05 04:21:40,274 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646454088477","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":11789,"client":"172.30.41.155:52176","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-05 04:21:40,266 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646454088477","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":11789,"client":"172.30.41.155:52176","multi.service_calls":0,"queuetimems":2,"class":"HRegionServer","multi.mutations":1} 2022-03-05 04:21:40,266 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646454088472","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":11794,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 04:21:40,568 INFO [Close-WAL-Writer-5] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646453905776 after 311ms 2022-03-05 04:21:45,920 INFO [Close-WAL-Writer-5] util.RecoverLeaseFSUtils: Recovered lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646453905776 after 5663ms 2022-03-05 04:21:45,942 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=289.55 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=15, accesses=3800, hits=3785, hitRatio=99.61%, , cachingAccesses=3800, cachingHits=3785, cachingHitsRatio=99.61%, evictions=298, evicted=0, evictedPerRun=0.0 2022-03-05 04:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 04:21:52,300 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646453927462.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646454110257.meta 2022-03-05 04:21:52,300 WARN [Close-WAL-Writer-5] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 04:21:52,300 WARN [Close-WAL-Writer-5] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 04:21:52,300 INFO [Close-WAL-Writer-5] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646453927462.meta 2022-03-05 04:21:52,301 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646453927462.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646453927462.meta 2022-03-05 04:21:52,316 INFO [Close-WAL-Writer-5] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646453927462.meta after 16ms 2022-03-05 04:21:56,318 WARN [Close-WAL-Writer-5] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646453927462.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646453927462.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 04:22:05,466 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1048ms No GCs detected 2022-03-05 04:22:06,648 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 8741 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:22:06,648 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 8540 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:22:06,648 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 8471 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:22:06,648 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 7895 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:22:06,648 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6952 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:22:06,685 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6917 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:22:06,685 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6291 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:22:06,685 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5972 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:22:06,685 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5443 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:22:06,685 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4837 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:22:06,685 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4488 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:22:06,685 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4488 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:22:06,685 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3888 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:22:08,917 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 615 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:22:09,869 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 04:22:11,676 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1084 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:22:14,263 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2086ms No GCs detected 2022-03-05 04:22:24,698 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1582 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:22:30,398 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6992 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:22:30,399 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 5200ms No GCs detected 2022-03-05 04:22:31,022 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 7378 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:22:35,512 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3913 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:22:35,512 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3379 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:22:35,512 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2241 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:22:35,512 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2150 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:22:35,528 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 731 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:22:49,016 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 960 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:22:58,182 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 511 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:23:01,427 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3021ms No GCs detected 2022-03-05 04:23:04,238 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1460ms No GCs detected 2022-03-05 04:23:04,261 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6079 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:23:04,261 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5929 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:23:04,261 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5501 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:23:04,261 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5349 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:23:04,261 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5096 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:23:04,404 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4686 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:23:04,404 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2983 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:23:04,404 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2981 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:23:25,607 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1029 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:23:25,607 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 850 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:23:31,545 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 908 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:23:31,545 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 746 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:23:31,545 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 611 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:23:31,545 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 574 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:23:41,909 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 593 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:23:54,316 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1171 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:23:56,437 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1506ms No GCs detected 2022-03-05 04:23:57,490 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3186 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:23:57,847 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2787 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:23:57,847 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2665 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:23:58,295 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 787 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:23:58,295 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 787 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:23:58,295 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 567 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:23:59,846 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 881 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:23:59,846 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 881 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:24:11,970 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 04:24:18,483 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2648 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:24:18,483 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2141 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:24:18,483 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1058 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:24:26,486 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646454110257.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646454266439.meta 2022-03-05 04:24:26,486 WARN [Close-WAL-Writer-6] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 04:24:26,486 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646454110257.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646454110257.meta 2022-03-05 04:24:26,487 WARN [Close-WAL-Writer-6] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 04:24:26,487 INFO [Close-WAL-Writer-6] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646454110257.meta 2022-03-05 04:24:26,517 INFO [Close-WAL-Writer-6] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646454110257.meta after 30ms 2022-03-05 04:24:30,545 WARN [Close-WAL-Writer-6] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646454110257.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646454110257.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 04:25:15,890 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 832 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:25:38,894 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1062 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:25:38,894 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 628 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=289.55 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=15, accesses=3822, hits=3807, hitRatio=99.61%, , cachingAccesses=3822, cachingHits=3807, cachingHitsRatio=99.61%, evictions=327, evicted=0, evictedPerRun=0.0 2022-03-05 04:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 04:28:23,359 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1008 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:28:24,112 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 687 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:28:30,726 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1043 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:28:30,727 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 873 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:28:30,734 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 513 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:29:08,370 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 509 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:29:23,695 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1114ms No GCs detected 2022-03-05 04:30:16,972 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3717ms No GCs detected 2022-03-05 04:30:21,274 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3802ms No GCs detected 2022-03-05 04:30:21,278 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 04:30:21,358 INFO [regionserver/cvp328:16201.Chore.1] hbase.ScheduledChore: Chore: MemstoreFlusherChore missed its start time 2022-03-05 04:30:21,358 INFO [regionserver/cvp328:16201.Chore.1] hbase.ScheduledChore: Chore: CompactionChecker missed its start time 2022-03-05 04:30:22,507 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 04:30:25,126 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2118ms No GCs detected 2022-03-05 04:30:26,193 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 04:30:26,193 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.sync(AsyncProtobufLogWriter.java:148) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:394) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.consume(AsyncFSWAL.java:581) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 04:30:26,268 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: rg.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 04:30:26,286 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 1 failover attempts. Trying to failover after sleeping for 503ms. 2022-03-05 04:30:26,789 WARN [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: A failover has occurred since the start of call #859 ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 2022-03-05 04:30:27,210 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:568) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.addBlock over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 04:30:27,212 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 2 failover attempts. Trying to failover after sleeping for 718ms. 2022-03-05 04:30:27,226 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:568) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.addBlock over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 839ms. 2022-03-05 04:30:27,938 WARN [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: A failover has occurred since the start of call #859 ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 2022-03-05 04:30:28,024 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: tNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 3 failover attempts. Trying to failover after sleeping for 2286ms. 2022-03-05 04:30:28,582 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: lientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:568) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.addBlock over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 2 failover attempts. Trying to failover after sleeping for 1086ms. 2022-03-05 04:30:29,670 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: lientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:568) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.addBlock over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 3 failover attempts. Trying to failover after sleeping for 3486ms. 2022-03-05 04:30:30,310 WARN [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: A failover has occurred since the start of call #859 ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 2022-03-05 04:30:30,312 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: tNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 4 failover attempts. Trying to failover after sleeping for 6701ms. 2022-03-05 04:30:33,176 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646454098957 with entries=1160, filesize=510.56 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646454626220 2022-03-05 04:30:33,176 WARN [Close-WAL-Writer-6] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 04:30:33,177 WARN [Close-WAL-Writer-6] wal.AsyncProtobufLogWriter: normal close failed, try recover java.lang.IllegalStateException: should call flush first before calling close at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:510) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:514) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 04:30:33,177 INFO [Close-WAL-Writer-6] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646454098957 2022-03-05 04:30:33,178 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646454621277","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":11901,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-05 04:30:33,179 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646454621797","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":11381,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":521,"class":"HRegionServer","multi.mutations":1} 2022-03-05 04:30:33,179 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646454622505","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":10673,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 04:30:33,220 INFO [Close-WAL-Writer-6] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646454098957 after 43ms 2022-03-05 04:30:37,014 WARN [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: A failover has occurred since the start of call #859 ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 2022-03-05 04:30:37,021 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: tNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 5 failover attempts. Trying to failover after sleeping for 5020ms. 2022-03-05 04:30:38,950 INFO [Close-WAL-Writer-6] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646454098957 after 5773ms 2022-03-05 04:30:39,957 INFO [Close-WAL-Writer-6] retry.RetryInvocationHandler: apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.isFileClosed(ClientNamenodeProtocolServerSideTranslatorPB.java:1322) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.isFileClosed over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 04:30:42,041 WARN [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: A failover has occurred since the start of call #859 ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 2022-03-05 04:30:49,985 INFO [Close-WAL-Writer-6] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=2 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646454098957 after 16808ms 2022-03-05 04:31:15,176 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 19918 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:31:18,513 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Requesting log roll because we exceeded slow sync threshold; time=19918 ms, threshold=10000 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:31:18,513 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.sync(AsyncProtobufLogWriter.java:148) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:394) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.consume(AsyncFSWAL.java:581) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 04:31:15,176 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646454655258","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":19918,"client":"172.30.41.155:52176","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-05 04:31:19,018 INFO [regionserver/cvp328:16201.Chore.2] hbase.ScheduledChore: Chore: CompactionChecker missed its start time 2022-03-05 04:31:17,726 WARN [regionserver/cvp328:16201] util.Sleeper: We slept 26468ms instead of 5000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired 2022-03-05 04:31:19,001 WARN [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 23240ms No GCs detected 2022-03-05 04:31:18,929 INFO [regionserver/cvp328:16201.Chore.1] hbase.ScheduledChore: Chore: MemstoreFlusherChore missed its start time 2022-03-05 04:31:23,911 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646454266439.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646454674945.meta 2022-03-05 04:31:23,911 WARN [Close-WAL-Writer-7] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 04:31:23,911 WARN [Close-WAL-Writer-7] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 04:31:23,918 INFO [Close-WAL-Writer-7] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646454266439.meta 2022-03-05 04:31:23,913 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646454266439.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646454266439.meta 2022-03-05 04:31:27,081 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1465ms No GCs detected 2022-03-05 04:31:27,208 INFO [Close-WAL-Writer-7] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646454266439.meta after 3290ms 2022-03-05 04:31:27,365 WARN [Close-WAL-Writer-6] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 04:31:27,365 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646454626220 with entries=88, filesize=40.20 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646454683911 2022-03-05 04:31:27,365 WARN [Close-WAL-Writer-6] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 04:31:27,366 INFO [Close-WAL-Writer-6] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646454626220 2022-03-05 04:31:27,392 INFO [Close-WAL-Writer-6] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646454626220 after 26ms 2022-03-05 04:31:31,210 WARN [Close-WAL-Writer-7] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646454266439.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646454266439.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 04:31:31,395 INFO [Close-WAL-Writer-6] util.RecoverLeaseFSUtils: Recovered lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646454626220 after 4029ms 2022-03-05 04:31:44,750 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-05 04:31:44,750 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=289.55 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=15, accesses=3850, hits=3835, hitRatio=99.61%, , cachingAccesses=3850, cachingHits=3835, cachingHitsRatio=99.61%, evictions=355, evicted=0, evictedPerRun=0.0 2022-03-05 04:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 04:31:58,962 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e. because 4333b489291b1f77671b8f2e78779a1e/__status_family___ has an old edit so flush to free WALs after random delay 162130 ms 2022-03-05 04:32:08,660 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 770 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:32:41,590 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 794 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:32:43,554 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1548 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:32:46,924 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1888 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:32:46,924 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1829 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:32:46,924 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1223 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:32:46,924 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 570 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:33:02,357 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1154ms No GCs detected 2022-03-05 04:33:08,140 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 5283ms No GCs detected 2022-03-05 04:33:10,095 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2201 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:33:10,095 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1951 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:33:10,099 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 804 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:33:10,099 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 804 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:33:10,120 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 683 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:33:10,120 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 682 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:33:10,120 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 682 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:33:10,120 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 678 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:33:21,555 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6549 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:33:21,555 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6484 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:33:21,555 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4913 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:33:21,564 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: rg.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 04:33:21,566 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 04:33:21,566 WARN [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: A failover has occurred since the start of call #895 ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 2022-03-05 04:33:21,837 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646454674945.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646454798444.meta 2022-03-05 04:33:21,838 WARN [Close-WAL-Writer-8] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 04:33:21,838 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646454674945.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646454674945.meta 2022-03-05 04:33:21,838 WARN [Close-WAL-Writer-8] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 04:33:21,838 INFO [Close-WAL-Writer-8] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646454674945.meta 2022-03-05 04:33:21,888 INFO [Close-WAL-Writer-8] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646454674945.meta after 50ms 2022-03-05 04:33:25,890 WARN [Close-WAL-Writer-8] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646454674945.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646454674945.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 04:33:29,637 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2091 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:33:39,210 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1190 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:33:39,379 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1164 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:34:13,563 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1467 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:34:13,573 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 711 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:34:41,094 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 4333b489291b1f77671b8f2e78779a1e 1/1 column families, dataSize=16.36 KB heapSize=25.84 KB 2022-03-05 04:34:41,175 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=16.36 KB at sequenceid=135 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/.tmp/__status_family___/5ae68257a6de4ac9a050e260e638642c 2022-03-05 04:34:41,186 INFO [MemStoreFlusher.0] retry.RetryInvocationHandler: doop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:433) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getBlockLocations over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 04:34:41,203 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/5ae68257a6de4ac9a050e260e638642c, entries=3, sequenceid=135, filesize=5.1 K 2022-03-05 04:34:41,206 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~16.36 KB/16752, heapSize ~25.82 KB/26440, currentSize=0 B/0 for 4333b489291b1f77671b8f2e78779a1e in 112ms, sequenceid=135, compaction requested=false 2022-03-05 04:34:59,350 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 646 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:35:18,567 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 734 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:35:18,567 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 543 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:35:19,713 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb. because 9b121c22e18472616debd0996ee2a4cb/e has an old edit so flush to free WALs after random delay 46172 ms 2022-03-05 04:35:19,713 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df. because 80a1ffe1dc2d8f258a6367d204dc64df/e has an old edit so flush to free WALs after random delay 2181 ms 2022-03-05 04:35:19,713 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391. because 0a764e51258807790269e46c55b99391/e has an old edit so flush to free WALs after random delay 183466 ms 2022-03-05 04:35:19,713 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5. because 7534b3b1812808d2d3c16342fbf347a5/e has an old edit so flush to free WALs after random delay 54846 ms 2022-03-05 04:35:19,713 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. because 7d1b53259f1d8280a47f7704f98f2c9f/e has an old edit so flush to free WALs after random delay 1380 ms 2022-03-05 04:35:19,713 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb. because ccdacf4b414cb8d1bc548efc77c663cb/e has an old edit so flush to free WALs after random delay 109993 ms 2022-03-05 04:35:21,094 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 7d1b53259f1d8280a47f7704f98f2c9f 1/1 column families, dataSize=1.03 MB heapSize=1.63 MB 2022-03-05 04:35:21,895 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 80a1ffe1dc2d8f258a6367d204dc64df 1/1 column families, dataSize=739.43 KB heapSize=1.17 MB 2022-03-05 04:35:25,118 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=739.43 KB at sequenceid=2126 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/.tmp/e/e6b6a1b3ae30483e860c86f3c706176b 2022-03-05 04:35:25,296 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=1.03 MB at sequenceid=2249 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/.tmp/e/7e4c998041da4bfba433015b584218e4 2022-03-05 04:35:25,309 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/e6b6a1b3ae30483e860c86f3c706176b, entries=5864, sequenceid=2126, filesize=104.4 K 2022-03-05 04:35:25,328 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~739.43 KB/757177, heapSize ~1.17 MB/1228680, currentSize=0 B/0 for 80a1ffe1dc2d8f258a6367d204dc64df in 3433ms, sequenceid=2126, compaction requested=false 2022-03-05 04:35:25,429 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/7e4c998041da4bfba433015b584218e4, entries=7823, sequenceid=2249, filesize=129.4 K 2022-03-05 04:35:25,439 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~1.03 MB/1077446, heapSize ~1.63 MB/1708624, currentSize=1.22 KB/1253 for 7d1b53259f1d8280a47f7704f98f2c9f in 4345ms, sequenceid=2249, compaction requested=false 2022-03-05 04:35:38,804 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3233 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:35:38,807 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1691ms No GCs detected 2022-03-05 04:35:40,973 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1164ms No GCs detected 2022-03-05 04:35:41,041 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2238 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:35:41,041 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1970 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:35:59,722 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1669ms No GCs detected 2022-03-05 04:36:09,388 INFO [regionserver/cvp328:16201.Chore.3] hbase.ScheduledChore: Chore: CompactionChecker missed its start time 2022-03-05 04:36:09,380 INFO [regionserver/cvp328:16201.Chore.2] hbase.ScheduledChore: Chore: MemstoreFlusherChore missed its start time 2022-03-05 04:36:09,767 WARN [regionserver/cvp328:16201] util.Sleeper: We slept 15699ms instead of 5000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired 2022-03-05 04:36:09,771 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.sync(AsyncProtobufLogWriter.java:148) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:394) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.consume(AsyncFSWAL.java:581) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 04:36:09,794 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 9b121c22e18472616debd0996ee2a4cb 1/1 column families, dataSize=911.21 KB heapSize=1.59 MB 2022-03-05 04:36:10,137 WARN [Close-WAL-Writer-7] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 04:36:10,137 WARN [Close-WAL-Writer-7] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 04:36:10,138 INFO [Close-WAL-Writer-7] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646454683911 2022-03-05 04:36:10,137 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646454683911 with entries=660, filesize=293.27 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646454969772 2022-03-05 04:36:10,143 INFO [Close-WAL-Writer-7] retry.RetryInvocationHandler: .apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.recoverLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 04:36:10,168 INFO [Close-WAL-Writer-7] retry.RetryInvocationHandler: menodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.recoverLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 336ms. 2022-03-05 04:36:10,187 INFO [MemStoreFlusher.1] retry.RetryInvocationHandler: rg.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 04:36:10,201 INFO [MemStoreFlusher.1] retry.RetryInvocationHandler: NamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 1 failover attempts. Trying to failover after sleeping for 638ms. 2022-03-05 04:36:10,504 WARN [Close-WAL-Writer-7] retry.RetryInvocationHandler: A failover has occurred since the start of call #976 ClientNamenodeProtocolTranslatorPB.recoverLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 2022-03-05 04:36:10,756 INFO [Close-WAL-Writer-7] retry.RetryInvocationHandler: enodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.recoverLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 2 failover attempts. Trying to failover after sleeping for 1227ms. 2022-03-05 04:36:11,983 WARN [Close-WAL-Writer-7] retry.RetryInvocationHandler: A failover has occurred since the start of call #976 ClientNamenodeProtocolTranslatorPB.recoverLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 2022-03-05 04:36:12,118 INFO [Close-WAL-Writer-7] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646454683911 after 1980ms 2022-03-05 04:36:12,164 INFO [MemStoreFlusher.1] retry.RetryInvocationHandler: rg.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 04:36:12,167 INFO [MemStoreFlusher.1] retry.RetryInvocationHandler: NamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 1 failover attempts. Trying to failover after sleeping for 312ms. 2022-03-05 04:36:12,480 INFO [MemStoreFlusher.1] retry.RetryInvocationHandler: NamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 2 failover attempts. Trying to failover after sleeping for 606ms. 2022-03-05 04:36:14,779 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 7534b3b1812808d2d3c16342fbf347a5 1/1 column families, dataSize=428.20 KB heapSize=744.40 KB 2022-03-05 04:36:15,670 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 887 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:36:15,676 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 590 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:36:15,676 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 570 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:36:15,940 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=911.61 KB at sequenceid=2453 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/.tmp/e/2dc27d135e10443d85b6c02899a10066 2022-03-05 04:36:16,033 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/2dc27d135e10443d85b6c02899a10066, entries=9490, sequenceid=2453, filesize=121 K 2022-03-05 04:36:16,038 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~911.61 KB/933491, heapSize ~1.59 MB/1664648, currentSize=2.98 KB/3055 for 9b121c22e18472616debd0996ee2a4cb in 6312ms, sequenceid=2453, compaction requested=false 2022-03-05 04:36:16,124 INFO [Close-WAL-Writer-7] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646454683911 after 5986ms 2022-03-05 04:36:16,318 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=428.20 KB at sequenceid=1576 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/.tmp/e/df10043559a04b41a80b794f69d00508 2022-03-05 04:36:16,344 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/df10043559a04b41a80b794f69d00508, entries=4034, sequenceid=1576, filesize=68.3 K 2022-03-05 04:36:16,346 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~428.20 KB/438477, heapSize ~744.38 KB/762248, currentSize=600 B/600 for 7534b3b1812808d2d3c16342fbf347a5 in 1567ms, sequenceid=1576, compaction requested=false 2022-03-05 04:36:30,453 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646454798444.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646454990156.meta 2022-03-05 04:36:30,453 WARN [Close-WAL-Writer-9] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 04:36:30,454 WARN [Close-WAL-Writer-9] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 04:36:30,454 INFO [Close-WAL-Writer-9] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646454798444.meta 2022-03-05 04:36:30,453 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646454798444.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646454798444.meta 2022-03-05 04:36:30,811 INFO [Close-WAL-Writer-9] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646454798444.meta after 357ms 2022-03-05 04:36:34,824 WARN [Close-WAL-Writer-9] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646454798444.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646454798444.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 04:36:44,269 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2205 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:36:44,269 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1369 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=289.55 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=15, accesses=3894, hits=3879, hitRatio=99.61%, , cachingAccesses=3894, cachingHits=3879, cachingHitsRatio=99.61%, evictions=384, evicted=0, evictedPerRun=0.0 2022-03-05 04:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 04:37:06,640 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3443ms No GCs detected 2022-03-05 04:37:06,682 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3517 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:37:06,682 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3187 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:37:06,682 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2020 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:37:09,707 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing ccdacf4b414cb8d1bc548efc77c663cb 1/1 column families, dataSize=639.19 KB heapSize=990.55 KB 2022-03-05 04:37:09,949 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=639.19 KB at sequenceid=1176 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/.tmp/e/4dc4fc5a78d64a1194c0d9cc47286201 2022-03-05 04:37:09,986 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/4dc4fc5a78d64a1194c0d9cc47286201, entries=4629, sequenceid=1176, filesize=98.3 K 2022-03-05 04:37:09,988 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~639.19 KB/654533, heapSize ~990.54 KB/1014312, currentSize=0 B/0 for ccdacf4b414cb8d1bc548efc77c663cb in 281ms, sequenceid=1176, compaction requested=false 2022-03-05 04:38:01,110 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1439ms No GCs detected 2022-03-05 04:38:01,169 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 625 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:38:19,442 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1707 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:38:19,442 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1339 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:38:21,466 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1575ms No GCs detected 2022-03-05 04:38:26,840 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2742ms No GCs detected 2022-03-05 04:38:26,846 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 0a764e51258807790269e46c55b99391 1/1 column families, dataSize=1.30 MB heapSize=2.16 MB 2022-03-05 04:38:29,542 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=1.30 MB at sequenceid=2677 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/.tmp/e/5e887b8fd93849e2983cc4ce3833c80d 2022-03-05 04:38:29,568 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/5e887b8fd93849e2983cc4ce3833c80d, entries=11883, sequenceid=2677, filesize=153.5 K 2022-03-05 04:38:29,570 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~1.30 MB/1361710, heapSize ~2.17 MB/2270616, currentSize=209 B/209 for 0a764e51258807790269e46c55b99391 in 2730ms, sequenceid=2677, compaction requested=false 2022-03-05 04:38:32,805 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 614 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:38:51,815 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2152ms No GCs detected 2022-03-05 04:39:24,087 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1466ms No GCs detected 2022-03-05 04:39:30,949 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1295ms No GCs detected 2022-03-05 04:39:32,777 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1328ms No GCs detected 2022-03-05 04:39:48,191 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3609ms No GCs detected 2022-03-05 04:39:48,224 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.MetricsWAL: AsyncFSWAL-0-hdfs://mycluster/hbase took 2995 ms appending an edit to wal; len~=203 2022-03-05 04:40:16,431 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1365 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:40:16,431 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1364 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:40:18,351 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1252ms No GCs detected 2022-03-05 04:40:19,574 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3230 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:40:19,574 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3176 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:40:43,919 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1315 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:40:43,919 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1315 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:40:43,919 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1019 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:41:07,141 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 607 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:41:15,263 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 582 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=289.55 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=15, accesses=3916, hits=3901, hitRatio=99.62%, , cachingAccesses=3916, cachingHits=3901, cachingHitsRatio=99.62%, evictions=414, evicted=0, evictedPerRun=0.0 2022-03-05 04:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 04:42:59,416 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 757 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:45:28,720 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1244 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:45:28,720 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 868 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:45:28,773 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 913 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:45:28,773 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 699 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:45:52,951 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 650 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:46:03,978 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2751 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:46:03,978 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2607 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:46:03,978 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1784 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:46:03,978 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1784 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:46:03,978 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1223 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:46:03,978 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 928 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:46:03,978 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 775 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:46:03,978 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 654 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:46:14,055 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646454969772 with entries=1354, filesize=598.73 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646455573147 2022-03-05 04:46:14,056 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646451113069 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646451113069 2022-03-05 04:46:14,103 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646451718029 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646451718029 2022-03-05 04:46:14,145 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646452318081 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646452318081 2022-03-05 04:46:14,215 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646452918125 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646452918125 2022-03-05 04:46:14,266 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646453282918 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646453282918 2022-03-05 04:46:14,286 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646453300668 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646453300668 2022-03-05 04:46:14,307 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646453905776 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646453905776 2022-03-05 04:46:14,320 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646454098957 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646454098957 2022-03-05 04:46:14,344 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646454626220 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646454626220 2022-03-05 04:46:25,884 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2314ms No GCs detected 2022-03-05 04:46:28,606 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1597 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 04:46:32,054 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5044 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 04:46:32,054 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5044 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 04:46:32,054 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5044 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 04:46:32,054 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5042 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 04:46:32,054 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4607 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 04:46:32,054 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4124 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 04:46:32,054 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3564 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 04:46:30,927 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1884ms No GCs detected 2022-03-05 04:46:38,256 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3590ms No GCs detected 2022-03-05 04:46:40,309 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 04:46:42,435 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 04:46:42,435 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 04:46:42,435 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 04:46:42,435 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 04:46:42,434 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 04:46:40,312 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1555ms No GCs detected 2022-03-05 04:46:43,394 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: NamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 828ms. 2022-03-05 04:46:44,437 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=307.84 KB, freeSize=306.90 MB, max=307.20 MB, blockCount=17, accesses=3941, hits=3924, hitRatio=99.57%, , cachingAccesses=3941, cachingHits=3924, cachingHitsRatio=99.57%, evictions=443, evicted=0, evictedPerRun=0.0 2022-03-05 04:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 04:46:51,606 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: 803) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:469) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.create over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 04:46:51,758 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646454990156.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646455598249.meta 2022-03-05 04:46:51,758 WARN [Close-WAL-Writer-10] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 04:46:51,758 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646454990156.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646454990156.meta 2022-03-05 04:46:51,759 WARN [Close-WAL-Writer-10] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 04:46:51,759 INFO [Close-WAL-Writer-10] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646454990156.meta 2022-03-05 04:46:51,767 INFO [Close-WAL-Writer-10] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646454990156.meta after 8ms 2022-03-05 04:46:51,804 WARN [Close-WAL-Writer-8] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 04:46:51,804 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646455573147 with entries=35, filesize=18.38 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646455611758 2022-03-05 04:46:51,805 WARN [Close-WAL-Writer-8] wal.AsyncProtobufLogWriter: normal close failed, try recover java.lang.IllegalStateException: should call flush first before calling close at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:510) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:514) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 04:46:51,805 INFO [Close-WAL-Writer-8] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646455573147 2022-03-05 04:46:51,808 INFO [Close-WAL-Writer-8] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646455573147 after 3ms 2022-03-05 04:46:51,809 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646455588908","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":22900,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 04:46:51,809 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646455588908","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":22900,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 04:46:51,809 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646455598250","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":13559,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":2441,"class":"HRegionServer","multi.mutations":1} 2022-03-05 04:46:51,810 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646455592053","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":19757,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":1131,"class":"HRegionServer","multi.mutations":1} 2022-03-05 04:46:51,810 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646455598269","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":13540,"client":"172.30.41.155:52176","multi.service_calls":0,"queuetimems":18,"class":"HRegionServer","multi.mutations":1} 2022-03-05 04:46:51,810 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646455591203","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":20606,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 04:46:51,811 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646455598290","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":13519,"client":"172.30.41.155:52176","multi.service_calls":0,"queuetimems":2482,"class":"HRegionServer","multi.mutations":1} 2022-03-05 04:46:51,810 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646455588606","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":23202,"client":"172.30.41.155:52176","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 04:46:51,809 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646455598291","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":13518,"client":"172.30.41.155:52176","multi.service_calls":0,"queuetimems":2484,"class":"HRegionServer","multi.mutations":1} 2022-03-05 04:46:51,813 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646455595808","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":16003,"client":"172.30.41.155:52176","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-05 04:46:51,812 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646455590950","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":20859,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":28,"class":"HRegionServer","multi.mutations":1} 2022-03-05 04:46:51,810 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646455598263","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":13547,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":9,"class":"HRegionServer","multi.mutations":1} 2022-03-05 04:46:51,814 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646455590923","responsesize":"18","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":20890,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":2} 2022-03-05 04:46:51,825 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646455598256","responsesize":"90","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 13 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x92\\xC4\\x06Kernel ","processingtimems":13569,"client":"172.31.0.219:42758","multi.service_calls":0,"queuetimems":2448,"class":"HRegionServer","multi.mutations":13} 2022-03-05 04:46:55,768 WARN [Close-WAL-Writer-10] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646454990156.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646454990156.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 04:46:55,809 INFO [Close-WAL-Writer-8] util.RecoverLeaseFSUtils: Recovered lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646455573147 after 4004ms 2022-03-05 04:46:58,748 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1075 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:46:58,748 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 804 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:46:58,748 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 676 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:47:13,165 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1099ms No GCs detected 2022-03-05 04:47:13,189 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4724 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:47:14,238 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4951 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:47:14,238 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4506 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:47:14,238 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2675 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:47:14,238 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2675 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:47:14,238 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1055 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:47:14,238 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1049 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:47:16,715 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 581 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:47:19,209 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 04:47:24,976 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2673 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:47:24,977 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1840 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:47:24,977 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1729 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:47:33,364 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2690 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:47:33,364 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2554 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:47:33,364 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2375 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:47:33,364 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2116 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:47:33,364 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1987 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:47:33,364 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1168 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:47:33,364 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 554 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:47:33,364 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 554 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:47:41,305 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 705 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:47:48,221 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1617ms No GCs detected 2022-03-05 04:47:48,499 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2751 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:48:13,378 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1578 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:48:13,378 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1306 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:48:32,215 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3546 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:48:32,215 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2498 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:48:32,216 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2484 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:48:32,216 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2021 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:48:32,216 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1808 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:48:32,216 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1444 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:48:32,216 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 941 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:48:32,216 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 657 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:48:32,216 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 625 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:48:45,048 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1884ms No GCs detected 2022-03-05 04:48:45,145 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2179 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:49:10,665 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 5152ms No GCs detected 2022-03-05 04:49:10,668 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6521 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:49:12,591 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 508 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:49:13,471 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 558 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:49:25,704 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1483ms No GCs detected 2022-03-05 04:49:32,881 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 6676ms No GCs detected 2022-03-05 04:49:32,961 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 9734 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:49:32,961 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 9677 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:49:33,188 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646455598249.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646455765701.meta 2022-03-05 04:49:33,189 WARN [Close-WAL-Writer-11] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 04:49:33,189 WARN [Close-WAL-Writer-11] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 04:49:33,189 INFO [Close-WAL-Writer-11] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646455598249.meta 2022-03-05 04:49:33,190 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646455598249.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646455598249.meta 2022-03-05 04:49:33,241 INFO [Close-WAL-Writer-11] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646455598249.meta after 52ms 2022-03-05 04:49:37,252 WARN [Close-WAL-Writer-11] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646455598249.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646455598249.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 04:49:41,102 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1714ms No GCs detected 2022-03-05 04:49:44,635 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2532ms No GCs detected 2022-03-05 04:49:45,431 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 788 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:49:48,649 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1271 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:49:48,649 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 621 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:49:51,648 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 04:50:50,752 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1448ms No GCs detected 2022-03-05 04:50:51,969 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 04:51:02,226 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 894 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=307.84 KB, freeSize=306.90 MB, max=307.20 MB, blockCount=17, accesses=3966, hits=3949, hitRatio=99.57%, , cachingAccesses=3966, cachingHits=3949, cachingHitsRatio=99.57%, evictions=472, evicted=0, evictedPerRun=0.0 2022-03-05 04:51:47,164 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 801 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 04:54:06,771 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1151 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 04:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=307.84 KB, freeSize=306.90 MB, max=307.20 MB, blockCount=17, accesses=3988, hits=3971, hitRatio=99.57%, , cachingAccesses=3988, cachingHits=3971, cachingHitsRatio=99.57%, evictions=502, evicted=0, evictedPerRun=0.0 2022-03-05 04:56:44,689 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646455611758 with entries=1329, filesize=582.17 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646456204673 2022-03-05 04:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 04:59:01,337 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 670 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 04:59:01,337 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 584 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 04:59:14,238 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2152 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 04:59:14,243 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1388 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 04:59:24,456 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1135 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 04:59:24,456 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 762 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 04:59:36,341 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646455765701.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646456374694.meta 2022-03-05 04:59:36,344 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646455765701.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646455765701.meta 2022-03-05 04:59:57,863 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1199 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 04:59:57,863 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 734 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 04:59:58,699 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 646 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:00:55,827 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 05:00:55,837 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 880ms. 2022-03-05 05:01:33,072 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 860 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:01:33,073 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 861 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:01:41,841 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1100ms No GCs detected 2022-03-05 05:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=307.84 KB, freeSize=306.90 MB, max=307.20 MB, blockCount=17, accesses=4010, hits=3993, hitRatio=99.58%, , cachingAccesses=4010, cachingHits=3993, cachingHitsRatio=99.58%, evictions=532, evicted=0, evictedPerRun=0.0 2022-03-05 05:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 05:02:09,493 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1160 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:02:38,394 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 549 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:03:03,435 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2166 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:03:03,435 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2070 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:03:03,435 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1218 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:03:03,435 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1183 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:03:03,435 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 658 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:03:28,339 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 05:03:34,905 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1068 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:03:35,968 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1439 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:03:36,295 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1269ms No GCs detected 2022-03-05 05:04:12,058 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3232ms No GCs detected 2022-03-05 05:04:31,274 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1577 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:04:31,274 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1555 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:04:31,274 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1066 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:04:31,274 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 902 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:04:31,274 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 505 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:05:08,745 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1148ms No GCs detected 2022-03-05 05:05:18,230 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 562 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:05:37,060 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1175ms No GCs detected 2022-03-05 05:05:46,978 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 9417ms No GCs detected 2022-03-05 05:05:47,024 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5480 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:05:47,024 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3325 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:05:47,043 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2316 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:06:05,798 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 5644ms No GCs detected 2022-03-05 05:06:09,002 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2703ms No GCs detected 2022-03-05 05:06:09,034 WARN [regionserver/cvp328:16201] util.Sleeper: We slept 15329ms instead of 5000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired 2022-03-05 05:06:09,064 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:06:28,414 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:06:28,414 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:06:28,415 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:06:28,415 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:06:28,415 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:06:23,344 WARN [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 12743ms No GCs detected 2022-03-05 05:06:29,683 INFO [regionserver/cvp328:16201.Chore.1] hbase.ScheduledChore: Chore: MemstoreFlusherChore missed its start time 2022-03-05 05:06:29,683 INFO [regionserver/cvp328:16201.Chore.1] hbase.ScheduledChore: Chore: CompactionChecker missed its start time 2022-03-05 05:06:30,631 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 05:06:30,685 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: rg.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 05:06:30,685 WARN [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: A failover has occurred since the start of call #1183 ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 2022-03-05 05:06:30,722 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 1 failover attempts. Trying to failover after sleeping for 575ms. 2022-03-05 05:06:30,722 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: NamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 1 failover attempts. Trying to failover after sleeping for 481ms. 2022-03-05 05:06:31,303 WARN [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: A failover has occurred since the start of call #1182 ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 2022-03-05 05:06:39,358 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 6778ms No GCs detected 2022-03-05 05:06:41,458 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646456374694.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646456788423.meta 2022-03-05 05:06:41,459 WARN [Close-WAL-Writer-13] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:06:41,459 WARN [Close-WAL-Writer-13] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:06:41,460 INFO [Close-WAL-Writer-13] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646456374694.meta 2022-03-05 05:06:41,460 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646456374694.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646456374694.meta 2022-03-05 05:06:41,465 INFO [Close-WAL-Writer-13] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646456374694.meta after 5ms 2022-03-05 05:06:41,495 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:568) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.addBlock over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 05:06:41,499 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:568) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.addBlock over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 1 failover attempts. Trying to failover after sleeping for 733ms. 2022-03-05 05:06:42,234 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: lientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:568) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.addBlock over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 2 failover attempts. Trying to failover after sleeping for 1194ms. 2022-03-05 05:06:43,748 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646456204673 with entries=1202, filesize=524.07 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646456801459 2022-03-05 05:06:43,748 WARN [Close-WAL-Writer-10] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:06:43,749 WARN [Close-WAL-Writer-10] wal.AsyncProtobufLogWriter: normal close failed, try recover java.lang.IllegalStateException: should call flush first before calling close at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:510) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:514) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:06:43,749 INFO [Close-WAL-Writer-10] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646456204673 2022-03-05 05:06:43,750 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646456758004","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":45746,"client":"172.30.41.155:52176","multi.service_calls":0,"queuetimems":337,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:06:43,750 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646456758539","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":45211,"client":"172.30.41.155:52176","multi.service_calls":0,"queuetimems":269,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:06:43,750 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646456765799","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":37951,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:06:43,750 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646456756743","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":47007,"client":"172.30.41.155:52176","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:06:43,751 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646456765816","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":37934,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":17,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:06:43,752 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646456769049","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":34703,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":3202,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:06:43,752 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646456765817","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":37935,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":18,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:06:43,753 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646456765817","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":37936,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":18,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:06:43,753 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646456766015","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":37737,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":167,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:06:43,753 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646456765799","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":37953,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:06:43,753 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646456765868","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":37884,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":20,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:06:43,757 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646456769049","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":34704,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":3201,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:06:43,759 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646456765848","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":37909,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:06:43,760 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646456765847","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":37906,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:06:43,760 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646456768995","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":34758,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":3147,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:06:43,761 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646456766025","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":37728,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":178,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:06:44,026 INFO [Close-WAL-Writer-10] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646456204673 after 277ms 2022-03-05 05:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=307.84 KB, freeSize=306.90 MB, max=307.20 MB, blockCount=17, accesses=4064, hits=4047, hitRatio=99.58%, , cachingAccesses=4064, cachingHits=4047, cachingHitsRatio=99.58%, evictions=561, evicted=0, evictedPerRun=0.0 2022-03-05 05:06:45,837 WARN [Close-WAL-Writer-13] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646456374694.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646456374694.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 05:06:47,853 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1988 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:06:47,853 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1545 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:06:48,274 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 05:06:49,005 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1149 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:06:50,074 INFO [Close-WAL-Writer-10] util.RecoverLeaseFSUtils: Recovered lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646456204673 after 6325ms 2022-03-05 05:06:50,611 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1104ms No GCs detected 2022-03-05 05:06:50,074 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1798 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:06:51,583 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2567 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:07:02,009 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 754 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:07:02,009 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 663 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:07:02,799 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 977 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:07:02,799 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 642 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:07:06,912 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2707 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:07:15,537 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.sync(AsyncProtobufLogWriter.java:148) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:394) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.consume(AsyncFSWAL.java:581) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:07:15,538 WARN [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 10332ms No GCs detected 2022-03-05 05:07:15,540 INFO [regionserver/cvp328:16201.Chore.3] hbase.ScheduledChore: Chore: MemstoreFlusherChore missed its start time 2022-03-05 05:07:15,540 INFO [regionserver/cvp328:16201.Chore.3] hbase.ScheduledChore: Chore: CompactionChecker missed its start time 2022-03-05 05:07:18,072 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 05:07:18,156 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2117ms No GCs detected 2022-03-05 05:07:18,221 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 05:07:18,221 WARN [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: A failover has occurred since the start of call #1196 ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 2022-03-05 05:07:20,969 WARN [Close-WAL-Writer-10] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:07:20,970 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646456801459 with entries=165, filesize=67.14 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646456835538 2022-03-05 05:07:20,970 WARN [Close-WAL-Writer-10] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:07:20,972 INFO [Close-WAL-Writer-10] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646456801459 2022-03-05 05:07:20,973 INFO [Close-WAL-Writer-10] retry.RetryInvocationHandler: .apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.recoverLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 05:07:21,261 INFO [Close-WAL-Writer-10] retry.RetryInvocationHandler: menodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.recoverLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 1 failover attempts. Trying to failover after sleeping for 322ms. 2022-03-05 05:07:21,599 INFO [Close-WAL-Writer-10] retry.RetryInvocationHandler: menodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.recoverLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 2 failover attempts. Trying to failover after sleeping for 835ms. 2022-03-05 05:07:27,409 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3241ms No GCs detected 2022-03-05 05:07:37,616 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 9704ms No GCs detected 2022-03-05 05:07:37,625 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 10212 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:07:37,799 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Requesting log roll because we exceeded slow sync threshold; time=10212 ms, threshold=10000 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:07:37,799 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.sync(AsyncProtobufLogWriter.java:148) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:394) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.consume(AsyncFSWAL.java:581) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:07:37,786 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646456847411","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":10375,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":2,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:07:37,787 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646456847409","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":10378,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:07:41,319 INFO [Close-WAL-Writer-10] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646456801459 after 20347ms 2022-03-05 05:07:42,774 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646456788423.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646456857456.meta 2022-03-05 05:07:42,777 WARN [Close-WAL-Writer-13] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:07:42,778 WARN [Close-WAL-Writer-13] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:07:42,778 INFO [Close-WAL-Writer-13] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646456788423.meta 2022-03-05 05:07:42,779 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646456788423.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646456788423.meta 2022-03-05 05:07:42,801 INFO [Close-WAL-Writer-13] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646456788423.meta after 23ms 2022-03-05 05:07:42,894 WARN [Close-WAL-Writer-11] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:07:42,894 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646456835538 with entries=39, filesize=20.24 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646456862777 2022-03-05 05:07:42,894 WARN [Close-WAL-Writer-11] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:07:42,895 INFO [Close-WAL-Writer-11] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646456835538 2022-03-05 05:07:42,915 INFO [Close-WAL-Writer-11] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646456835538 after 20ms 2022-03-05 05:07:45,751 INFO [Close-WAL-Writer-10] util.RecoverLeaseFSUtils: Recovered lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646456801459 after 24779ms 2022-03-05 05:07:46,804 WARN [Close-WAL-Writer-13] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646456788423.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646456788423.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 05:07:46,915 INFO [Close-WAL-Writer-11] util.RecoverLeaseFSUtils: Recovered lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646456835538 after 4020ms 2022-03-05 05:07:49,903 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1616ms No GCs detected 2022-03-05 05:07:49,927 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1893 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:07:54,864 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1952ms No GCs detected 2022-03-05 05:07:59,879 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3015ms No GCs detected 2022-03-05 05:08:06,754 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1370ms No GCs detected 2022-03-05 05:08:20,079 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 05:08:33,590 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 532 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:09:53,102 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1321ms No GCs detected 2022-03-05 05:10:12,046 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 852 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:10:20,866 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 6876ms No GCs detected 2022-03-05 05:10:28,768 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1391ms No GCs detected 2022-03-05 05:10:29,760 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2830 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:10:50,494 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: rg.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 05:10:51,376 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646456857456.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646457050491.meta 2022-03-05 05:10:51,376 WARN [Close-WAL-Writer-14] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:10:51,377 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646456857456.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646456857456.meta 2022-03-05 05:10:51,377 WARN [Close-WAL-Writer-14] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:10:51,377 INFO [Close-WAL-Writer-14] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646456857456.meta 2022-03-05 05:10:51,387 INFO [Close-WAL-Writer-14] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646456857456.meta after 10ms 2022-03-05 05:10:55,389 WARN [Close-WAL-Writer-14] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646456857456.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646456857456.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 05:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=307.84 KB, freeSize=306.90 MB, max=307.20 MB, blockCount=17, accesses=4086, hits=4069, hitRatio=99.58%, , cachingAccesses=4086, cachingHits=4069, cachingHitsRatio=99.58%, evictions=590, evicted=0, evictedPerRun=0.0 2022-03-05 05:11:48,202 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 05:14:00,561 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1992 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:14:00,561 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1832 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:14:00,561 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 933 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:14:00,561 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 801 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:14:28,757 INFO [regionserver/cvp328:16201.Chore.1] hbase.ScheduledChore: Chore: CompactionChecker missed its start time 2022-03-05 05:14:31,900 INFO [regionserver/cvp328:16201.Chore.3] hbase.ScheduledChore: Chore: MemstoreFlusherChore missed its start time 2022-03-05 05:14:31,901 WARN [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 15757ms No GCs detected 2022-03-05 05:14:31,878 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.sync(AsyncProtobufLogWriter.java:148) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:394) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.consume(AsyncFSWAL.java:581) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:14:31,802 WARN [regionserver/cvp328:16201] util.Sleeper: We slept 18657ms instead of 5000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired 2022-03-05 05:14:33,799 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646457050491.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646457271900.meta 2022-03-05 05:14:33,799 WARN [Close-WAL-Writer-15] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:14:33,799 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646457050491.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646457050491.meta 2022-03-05 05:14:33,799 WARN [Close-WAL-Writer-15] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:14:33,799 INFO [Close-WAL-Writer-15] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646457050491.meta 2022-03-05 05:14:33,811 INFO [Close-WAL-Writer-15] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646457050491.meta after 12ms 2022-03-05 05:14:33,847 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646456862777 with entries=898, filesize=387.76 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646457273799 2022-03-05 05:14:33,848 WARN [Close-WAL-Writer-12] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:14:33,850 WARN [Close-WAL-Writer-12] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:14:33,850 INFO [Close-WAL-Writer-12] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646456862777 2022-03-05 05:14:33,870 INFO [Close-WAL-Writer-12] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646456862777 after 20ms 2022-03-05 05:14:37,815 WARN [Close-WAL-Writer-15] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646457050491.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646457050491.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 05:14:44,608 INFO [Close-WAL-Writer-12] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646456862777 after 10758ms 2022-03-05 05:14:44,610 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6428 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:14:44,655 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5363 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:14:44,655 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4120 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:14:44,655 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3352 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:14:44,655 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2939 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:14:44,655 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2576 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:14:44,655 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1767 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:14:47,479 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2300 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:14:47,480 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 714 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:14:58,266 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 827 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:14:59,757 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 541 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:15:01,982 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1729ms No GCs detected 2022-03-05 05:15:05,814 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1071ms No GCs detected 2022-03-05 05:15:07,810 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1083 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:15:07,816 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 731 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:15:16,406 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5122 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:15:16,407 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4617 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:15:16,407 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4343 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:15:16,407 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3089 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:15:16,407 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1407 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:15:16,407 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1354 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:15:16,407 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 679 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:15:36,360 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 10320 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:15:36,360 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Requesting log roll because we exceeded slow sync threshold; time=10320 ms, threshold=10000 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:15:36,360 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 10108 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:15:36,360 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Requesting log roll because we exceeded slow sync threshold; time=10108 ms, threshold=10000 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:15:36,360 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 9673 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:15:36,360 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 8896 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:15:36,360 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 8636 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:15:36,360 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 8417 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:15:36,360 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 8307 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:15:36,360 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 7635 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:15:36,360 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 7635 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:15:36,361 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646457326251","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":10109,"client":"172.30.41.155:52176","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:15:36,362 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646457326039","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":10322,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:15:38,434 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646457328053","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":10381,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:15:38,452 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646457327943","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":10509,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:15:38,436 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646457326686","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":11749,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:15:38,435 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646457327723","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":10712,"client":"172.30.41.155:52176","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:15:40,869 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646457327463","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":10426,"client":"172.30.41.155:52176","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:15:40,912 WARN [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 11181ms No GCs detected 2022-03-05 05:15:40,920 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646457327723","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":13189,"client":"172.30.41.155:52176","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:15:40,920 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646457328724","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":12188,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:15:40,919 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Connection to 172.31.0.219/172.31.0.219:15010 closed at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$channelInactive$2(FanOutOneBlockAsyncDFSOutput.java:286) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.channelInactive(FanOutOneBlockAsyncDFSOutput.java:285) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:389) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:354) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelInactive(IdleStateHandler.java:277) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:831) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:15:40,933 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Connection to 172.31.0.219/172.31.0.219:15010 closed at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$channelInactive$2(FanOutOneBlockAsyncDFSOutput.java:286) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.channelInactive(FanOutOneBlockAsyncDFSOutput.java:285) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:389) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:354) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelInactive(IdleStateHandler.java:277) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:831) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:15:40,980 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 05:15:40,980 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: rg.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 05:15:40,980 WARN [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: A failover has occurred since the start of call #1248 ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 2022-03-05 05:15:41,183 WARN [Close-WAL-Writer-12] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:15:41,183 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646457273799 with entries=161, filesize=73.23 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646457338432 2022-03-05 05:15:41,184 WARN [Close-WAL-Writer-12] wal.AsyncProtobufLogWriter: normal close failed, try recover java.lang.IllegalStateException: should call flush first before calling close at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:510) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:514) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:15:41,185 INFO [Close-WAL-Writer-12] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646457273799 2022-03-05 05:15:41,242 INFO [Close-WAL-Writer-12] retry.RetryInvocationHandler: .apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.recoverLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 05:15:41,258 INFO [Close-WAL-Writer-12] retry.RetryInvocationHandler: menodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.recoverLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 1 failover attempts. Trying to failover after sleeping for 558ms. 2022-03-05 05:15:41,817 INFO [Close-WAL-Writer-12] retry.RetryInvocationHandler: menodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.recoverLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 2 failover attempts. Trying to failover after sleeping for 629ms. 2022-03-05 05:15:43,440 INFO [Close-WAL-Writer-12] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646457273799 after 2255ms 2022-03-05 05:15:47,625 INFO [Close-WAL-Writer-12] retry.RetryInvocationHandler: .apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.recoverLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 05:15:49,158 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1162 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:15:52,082 INFO [Close-WAL-Writer-12] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646457273799 after 10897ms 2022-03-05 05:15:53,799 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 842 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:16:02,735 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 578 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:16:12,761 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646457271900.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646457371386.meta 2022-03-05 05:16:12,761 WARN [Close-WAL-Writer-16] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:16:12,762 WARN [Close-WAL-Writer-16] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:16:12,762 INFO [Close-WAL-Writer-16] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646457271900.meta 2022-03-05 05:16:12,762 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646457271900.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646457271900.meta 2022-03-05 05:16:12,775 INFO [Close-WAL-Writer-16] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646457271900.meta after 13ms 2022-03-05 05:16:16,777 WARN [Close-WAL-Writer-16] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646457271900.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646457271900.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 05:16:45,866 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=307.84 KB, freeSize=306.90 MB, max=307.20 MB, blockCount=17, accesses=4108, hits=4091, hitRatio=99.59%, , cachingAccesses=4108, cachingHits=4091, cachingHitsRatio=99.59%, evictions=618, evicted=0, evictedPerRun=0.0 2022-03-05 05:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 05:17:02,779 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 668 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:17:02,779 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 668 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:18:53,364 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1285ms No GCs detected 2022-03-05 05:19:01,282 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1192ms No GCs detected 2022-03-05 05:19:13,605 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 05:20:01,847 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 925 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:20:02,569 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 721 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:20:11,741 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1168 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:20:34,841 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2078ms No GCs detected 2022-03-05 05:20:47,372 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1336 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:20:59,138 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 895 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:21:44,429 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=307.84 KB, freeSize=306.90 MB, max=307.20 MB, blockCount=17, accesses=4130, hits=4113, hitRatio=99.59%, , cachingAccesses=4130, cachingHits=4113, cachingHitsRatio=99.59%, evictions=648, evicted=0, evictedPerRun=0.0 2022-03-05 05:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 05:22:02,999 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1031ms No GCs detected 2022-03-05 05:22:18,155 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 700 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:22:27,069 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1047 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:22:27,069 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 693 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:22:27,962 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 521 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:23:38,550 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 689 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:24:59,190 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 543 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:24:59,190 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 543 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:25:33,912 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1144ms No GCs detected 2022-03-05 05:25:33,956 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1784 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:25:44,899 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646457338432 with entries=1328, filesize=583.66 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646457944859 2022-03-05 05:26:15,057 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646457371386.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646457974906.meta 2022-03-05 05:26:15,058 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646457371386.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646457371386.meta 2022-03-05 05:26:34,441 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 547 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=307.84 KB, freeSize=306.90 MB, max=307.20 MB, blockCount=17, accesses=4152, hits=4135, hitRatio=99.59%, , cachingAccesses=4152, cachingHits=4135, cachingHitsRatio=99.59%, evictions=678, evicted=0, evictedPerRun=0.0 2022-03-05 05:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 05:27:04,920 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 610 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:27:26,771 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1307 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:27:26,772 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 785 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:28:00,463 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 560 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:28:02,261 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 651 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:28:04,687 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 871 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:28:04,687 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 512 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:28:10,081 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 674 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:28:22,843 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 547 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:28:25,156 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1026ms No GCs detected 2022-03-05 05:28:25,222 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1760 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:28:35,289 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4545 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:28:35,290 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4522 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:28:35,290 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4350 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:28:35,301 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4045 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:28:35,301 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3968 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:28:35,301 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3115 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:28:35,301 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3115 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:28:35,301 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2513 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:28:35,301 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2513 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:28:35,301 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2513 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:28:35,301 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2269 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:28:35,301 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1118 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:28:42,969 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1540 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:28:50,977 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 6547ms No GCs detected 2022-03-05 05:28:50,998 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 05:28:50,979 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 8009 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:28:52,523 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 819 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:28:52,523 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 748 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:28:52,648 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 310ms. 2022-03-05 05:28:53,103 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: tNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 2 failover attempts. Trying to failover after sleeping for 1227ms. 2022-03-05 05:28:54,455 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: tNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 3 failover attempts. Trying to failover after sleeping for 2278ms. 2022-03-05 05:29:00,034 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1449ms No GCs detected 2022-03-05 05:29:16,073 WARN [regionserver/cvp328:16201] util.Sleeper: We slept 19357ms instead of 5000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired 2022-03-05 05:29:16,076 WARN [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 11147ms No GCs detected 2022-03-05 05:29:16,075 INFO [regionserver/cvp328:16201.Chore.1] hbase.ScheduledChore: Chore: CompactionChecker missed its start time 2022-03-05 05:29:21,632 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:29:21,688 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:29:21,688 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.sync(AsyncProtobufLogWriter.java:148) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:394) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.consume(AsyncFSWAL.java:581) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:29:21,932 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646457974906.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646458159813.meta 2022-03-05 05:29:21,933 WARN [Close-WAL-Writer-18] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:29:21,933 WARN [Close-WAL-Writer-18] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:29:21,933 INFO [Close-WAL-Writer-18] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646457974906.meta 2022-03-05 05:29:21,938 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646457974906.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646457974906.meta 2022-03-05 05:29:22,055 INFO [Close-WAL-Writer-18] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646457974906.meta after 122ms 2022-03-05 05:29:22,263 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646457944859 with entries=422, filesize=186.38 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646458161933 2022-03-05 05:29:22,264 WARN [Close-WAL-Writer-14] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:29:22,264 WARN [Close-WAL-Writer-14] wal.AsyncProtobufLogWriter: normal close failed, try recover java.lang.IllegalStateException: should call flush first before calling close at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:510) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:514) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:29:22,264 INFO [Close-WAL-Writer-14] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646457944859 2022-03-05 05:29:22,271 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646458140050","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":22221,"client":"172.30.41.155:52176","multi.service_calls":0,"queuetimems":16,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:29:22,272 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646458138187","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":24084,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:29:22,273 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646458140034","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":22237,"client":"172.30.41.155:52176","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:29:22,318 INFO [Close-WAL-Writer-14] retry.RetryInvocationHandler: .apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.recoverLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 05:29:22,326 INFO [Close-WAL-Writer-14] retry.RetryInvocationHandler: menodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.recoverLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 592ms. 2022-03-05 05:29:22,920 INFO [Close-WAL-Writer-14] retry.RetryInvocationHandler: enodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.recoverLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 2 failover attempts. Trying to failover after sleeping for 1275ms. 2022-03-05 05:29:24,379 INFO [Close-WAL-Writer-14] retry.RetryInvocationHandler: enodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.recoverLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 3 failover attempts. Trying to failover after sleeping for 1691ms. 2022-03-05 05:29:26,057 INFO [Close-WAL-Writer-18] retry.RetryInvocationHandler: .apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.recoverLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 05:29:26,062 WARN [Close-WAL-Writer-18] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646457974906.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646457974906.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 05:29:26,070 WARN [Close-WAL-Writer-14] retry.RetryInvocationHandler: A failover has occurred since the start of call #1307 ClientNamenodeProtocolTranslatorPB.recoverLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 2022-03-05 05:29:26,343 INFO [Close-WAL-Writer-14] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646457944859 after 4079ms 2022-03-05 05:29:30,344 INFO [Close-WAL-Writer-14] util.RecoverLeaseFSUtils: Recovered lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646457944859 after 8080ms 2022-03-05 05:29:42,160 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 659 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:29:43,844 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1684 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:29:44,043 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1048 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:29:56,086 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3877ms No GCs detected 2022-03-05 05:29:56,765 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:29:56,850 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:29:56,850 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:30:02,418 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:568) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.addBlock over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 05:30:02,441 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:568) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.addBlock over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 648ms. 2022-03-05 05:30:03,097 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: lientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:568) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.addBlock over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 2 failover attempts. Trying to failover after sleeping for 1108ms. 2022-03-05 05:30:09,782 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646458159813.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646458196707.meta 2022-03-05 05:30:09,782 WARN [Close-WAL-Writer-18] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:30:09,782 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646458159813.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646458159813.meta 2022-03-05 05:30:09,783 WARN [Close-WAL-Writer-18] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:30:09,783 INFO [Close-WAL-Writer-18] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646458159813.meta 2022-03-05 05:30:09,985 INFO [Close-WAL-Writer-18] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646458159813.meta after 202ms 2022-03-05 05:30:10,762 WARN [Close-WAL-Writer-14] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:30:10,762 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646458161933 with entries=114, filesize=48.04 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646458209782 2022-03-05 05:30:10,764 WARN [Close-WAL-Writer-14] wal.AsyncProtobufLogWriter: normal close failed, try recover java.lang.IllegalStateException: should call flush first before calling close at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:510) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:514) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:30:10,764 INFO [Close-WAL-Writer-14] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646458161933 2022-03-05 05:30:10,775 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646458196846","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":13929,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":35,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:30:10,775 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646458199760","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":11015,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:30:10,775 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646458200099","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":10676,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":339,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:30:10,776 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646458196892","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":13883,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":2,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:30:10,777 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646458196759","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":14016,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:30:10,777 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646458196846","responsesize":"18","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":13931,"client":"172.30.41.155:52176","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-05 05:30:10,775 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646458187399","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":23376,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:30:10,775 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646458196846","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":13929,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":35,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:30:10,777 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646458199760","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":11017,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:30:10,776 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646458198074","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":12702,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:30:10,776 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646458186314","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":24461,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:30:10,776 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646458196880","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":13895,"client":"172.30.41.155:52176","multi.service_calls":0,"queuetimems":34,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:30:10,775 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646458196846","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":13929,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":87,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:30:10,775 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646458188049","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":22726,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:30:10,783 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646458200013","responsesize":"18","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":10770,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":253,"class":"HRegionServer","multi.mutations":2} 2022-03-05 05:30:10,788 INFO [Close-WAL-Writer-14] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646458161933 after 24ms 2022-03-05 05:30:10,798 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646458196816","responsesize":"98","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 14 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x92\\xC4\\x06Kernel ","processingtimems":13982,"client":"172.30.41.155:52234","multi.service_calls":0,"queuetimems":5,"class":"HRegionServer","multi.mutations":14} 2022-03-05 05:30:13,987 WARN [Close-WAL-Writer-18] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646458159813.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646458159813.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 05:30:14,789 INFO [Close-WAL-Writer-14] util.RecoverLeaseFSUtils: Recovered lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646458161933 after 4025ms 2022-03-05 05:30:19,170 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1133 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:30:19,170 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1061 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:30:27,627 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 887 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:30:32,771 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 604 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:30:51,560 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2771ms No GCs detected 2022-03-05 05:31:01,140 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 05:31:11,065 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 539 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:31:16,345 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 634 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:31:28,428 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 728 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:31:28,428 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 687 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:31:28,428 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 559 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:31:38,603 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 691 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:31:41,969 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 513 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=307.84 KB, freeSize=306.90 MB, max=307.20 MB, blockCount=17, accesses=4446, hits=4429, hitRatio=99.62%, , cachingAccesses=4446, cachingHits=4429, cachingHitsRatio=99.62%, evictions=706, evicted=0, evictedPerRun=0.0 2022-03-05 05:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-05 05:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 05:32:00,930 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3514 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:32:00,931 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2921 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:32:00,931 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2512 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:32:00,931 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2210 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:32:00,933 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1265 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:32:00,933 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1174 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:32:08,396 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 538 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:32:14,197 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1191 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:32:25,133 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2775 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:32:25,133 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1995 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:32:31,650 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 05:32:59,842 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2368 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:32:59,842 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2007 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:32:59,842 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1955 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:32:59,956 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1635 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:32:59,956 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 987 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:33:06,372 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 729 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:33:12,422 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 500 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:33:37,327 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 14824 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:33:37,327 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Requesting log roll because we exceeded slow sync threshold; time=14824 ms, threshold=10000 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:33:37,327 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Connection to 172.31.0.219/172.31.0.219:15010 closed at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$channelInactive$2(FanOutOneBlockAsyncDFSOutput.java:286) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.channelInactive(FanOutOneBlockAsyncDFSOutput.java:285) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:389) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:354) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelInactive(IdleStateHandler.java:277) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:831) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:33:37,341 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.sync(AsyncProtobufLogWriter.java:148) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:394) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.consume(AsyncFSWAL.java:581) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:33:38,224 WARN [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 15227ms No GCs detected 2022-03-05 05:33:38,346 INFO [regionserver/cvp328:16201.Chore.3] hbase.ScheduledChore: Chore: MemstoreFlusherChore missed its start time 2022-03-05 05:33:38,389 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646458402497","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":15849,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:33:39,027 WARN [Close-WAL-Writer-19] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:33:39,027 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646458196707.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646458418033.meta 2022-03-05 05:33:39,027 WARN [Close-WAL-Writer-19] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:33:39,106 INFO [Close-WAL-Writer-19] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646458196707.meta 2022-03-05 05:33:39,106 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646458196707.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646458196707.meta 2022-03-05 05:33:39,391 INFO [Close-WAL-Writer-19] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646458196707.meta after 285ms 2022-03-05 05:33:39,958 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646458209782 with entries=475, filesize=212.17 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646458419105 2022-03-05 05:33:39,958 WARN [Close-WAL-Writer-15] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:33:39,958 WARN [Close-WAL-Writer-15] wal.AsyncProtobufLogWriter: normal close failed, try recover java.lang.IllegalStateException: should call flush first before calling close at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:510) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:514) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:33:39,958 INFO [Close-WAL-Writer-15] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646458209782 2022-03-05 05:33:39,962 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646458405911","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":14051,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":2409,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:33:39,962 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646458407515","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":12447,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:33:40,136 INFO [Close-WAL-Writer-15] retry.RetryInvocationHandler: .apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.recoverLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 05:33:40,146 INFO [Close-WAL-Writer-15] retry.RetryInvocationHandler: menodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.recoverLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 1 failover attempts. Trying to failover after sleeping for 737ms. 2022-03-05 05:33:40,887 INFO [Close-WAL-Writer-15] retry.RetryInvocationHandler: enodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.recoverLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 2 failover attempts. Trying to failover after sleeping for 1775ms. 2022-03-05 05:33:43,009 INFO [Close-WAL-Writer-15] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646458209782 after 3051ms 2022-03-05 05:33:43,392 WARN [Close-WAL-Writer-19] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646458196707.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646458196707.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 05:33:46,628 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 672 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:33:51,275 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5319 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:33:51,275 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4679 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:33:48,752 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1128ms No GCs detected 2022-03-05 05:33:51,945 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 611 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:33:57,344 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1383ms No GCs detected 2022-03-05 05:33:57,641 INFO [Close-WAL-Writer-15] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646458209782 after 17683ms 2022-03-05 05:33:57,717 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2256 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:33:57,718 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1711 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:33:57,718 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1042 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:33:59,146 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1212 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:33:59,146 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1028 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:33:59,792 INFO [Close-WAL-Writer-15] retry.RetryInvocationHandler: apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.isFileClosed(ClientNamenodeProtocolServerSideTranslatorPB.java:1322) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.isFileClosed over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 05:33:59,796 INFO [Close-WAL-Writer-15] retry.RetryInvocationHandler: enodeProtocolServerSideTranslatorPB.isFileClosed(ClientNamenodeProtocolServerSideTranslatorPB.java:1322) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.isFileClosed over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 536ms. 2022-03-05 05:34:00,385 INFO [Close-WAL-Writer-15] retry.RetryInvocationHandler: enodeProtocolServerSideTranslatorPB.isFileClosed(ClientNamenodeProtocolServerSideTranslatorPB.java:1322) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.isFileClosed over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 2 failover attempts. Trying to failover after sleeping for 606ms. 2022-03-05 05:34:31,015 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:34:33,563 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:34:33,563 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:34:33,563 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:34:33,563 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:34:33,456 WARN [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 18358ms No GCs detected 2022-03-05 05:34:33,408 INFO [regionserver/cvp328:16201.Chore.3] hbase.ScheduledChore: Chore: MemstoreFlusherChore missed its start time 2022-03-05 05:34:31,658 INFO [regionserver/cvp328:16201.Chore.2] hbase.ScheduledChore: Chore: CompactionChecker missed its start time 2022-03-05 05:34:36,444 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646458419105 with entries=124, filesize=53.06 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646458473563 2022-03-05 05:34:36,446 WARN [Close-WAL-Writer-15] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:34:36,446 WARN [Close-WAL-Writer-15] wal.AsyncProtobufLogWriter: normal close failed, try recover java.lang.IllegalStateException: should call flush first before calling close at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:510) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:514) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:34:36,446 INFO [Close-WAL-Writer-15] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646458419105 2022-03-05 05:34:36,452 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646458451855","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":24597,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:34:36,452 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646458453051","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":23401,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:34:36,453 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646458452199","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":24254,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":180,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:34:36,453 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646458457115","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":19338,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:34:36,453 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646458459183","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":17270,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:34:36,453 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646458457115","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":19338,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:34:36,475 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646458459181","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":17272,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":2066,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:34:36,505 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646458457115","responsesize":"108","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 16 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x92\\xC4\\x06Kernel ","processingtimems":19390,"client":"172.30.41.155:52234","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":16} 2022-03-05 05:34:36,556 INFO [Close-WAL-Writer-15] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646458419105 after 110ms 2022-03-05 05:34:40,030 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 673 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:34:40,810 INFO [Close-WAL-Writer-15] util.RecoverLeaseFSUtils: Recovered lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646458419105 after 4364ms 2022-03-05 05:34:43,054 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1602 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:34:43,054 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1062 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:34:43,054 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1055 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:34:47,526 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1436 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:34:48,041 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646458418033.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646458486525.meta 2022-03-05 05:34:48,041 WARN [Close-WAL-Writer-20] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:34:48,042 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646458418033.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646458418033.meta 2022-03-05 05:34:48,042 WARN [Close-WAL-Writer-20] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:34:48,042 INFO [Close-WAL-Writer-20] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646458418033.meta 2022-03-05 05:34:48,103 INFO [Close-WAL-Writer-20] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646458418033.meta after 61ms 2022-03-05 05:34:52,965 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 612 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:34:52,974 WARN [Close-WAL-Writer-20] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646458418033.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646458418033.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 05:34:56,512 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1500 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:34:58,162 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2697 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:34:58,162 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2161 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:34:58,162 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1649 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:34:58,162 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1434 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:34:59,089 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 921 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:35:01,092 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2448 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:35:01,092 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1431 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:35:01,092 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1150 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:35:02,665 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2266 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:35:04,934 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1770ms No GCs detected 2022-03-05 05:35:05,434 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4348 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:35:05,434 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4195 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:35:05,434 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3493 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:35:07,008 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1200 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:35:07,008 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 579 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:35:08,316 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1214 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:35:08,316 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1211 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:35:08,399 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 702 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:35:08,399 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 561 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:35:12,693 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 720 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:35:25,597 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2417 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:35:25,597 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2356 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:35:27,166 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1185 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:35:33,491 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df. because 80a1ffe1dc2d8f258a6367d204dc64df/e has an old edit so flush to free WALs after random delay 53754 ms 2022-03-05 05:35:33,491 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. because 7d1b53259f1d8280a47f7704f98f2c9f/e has an old edit so flush to free WALs after random delay 1577 ms 2022-03-05 05:35:33,867 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 570 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:35:35,069 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 7d1b53259f1d8280a47f7704f98f2c9f 1/1 column families, dataSize=643.75 KB heapSize=1.08 MB 2022-03-05 05:35:36,255 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=643.75 KB at sequenceid=3314 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/.tmp/e/a8ba5004ed1743a6a9ca49a0c0e55f2a 2022-03-05 05:35:36,347 INFO [MemStoreFlusher.0] retry.RetryInvocationHandler: doop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:433) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getBlockLocations over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 05:35:36,590 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/a8ba5004ed1743a6a9ca49a0c0e55f2a, entries=6220, sequenceid=3314, filesize=61.4 K 2022-03-05 05:35:36,609 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~643.75 KB/659204, heapSize ~1.08 MB/1131672, currentSize=0 B/0 for 7d1b53259f1d8280a47f7704f98f2c9f in 1540ms, sequenceid=3314, compaction requested=true 2022-03-05 05:35:36,620 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 7d1b53259f1d8280a47f7704f98f2c9f/e in aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. 2022-03-05 05:35:36,622 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/15f52eafa093478095f825d7f5718dd1, hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/7e4c998041da4bfba433015b584218e4, hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/a8ba5004ed1743a6a9ca49a0c0e55f2a] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/.tmp, totalSize=195.7 K 2022-03-05 05:35:37,030 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 7d1b53259f1d8280a47f7704f98f2c9f#e#compaction#13 average throughput is 13.91 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 05:35:39,966 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 673 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:35:40,392 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 7d1b53259f1d8280a47f7704f98f2c9f/e of 7d1b53259f1d8280a47f7704f98f2c9f into 6f863164283141d9a111eeda1fc65a11(size=195.7 K), total size for store is 195.7 K. This selection was in queue for 0sec, and took 3sec to execute. 2022-03-05 05:35:40,393 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., storeName=7d1b53259f1d8280a47f7704f98f2c9f/e, priority=13, startTime=1646458536611; duration=3sec 2022-03-05 05:35:53,438 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e. because 4333b489291b1f77671b8f2e78779a1e/__status_family___ has an old edit so flush to free WALs after random delay 31220 ms 2022-03-05 05:36:13,016 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1018 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:36:13,408 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb. because 9b121c22e18472616debd0996ee2a4cb/e has an old edit so flush to free WALs after random delay 68926 ms 2022-03-05 05:36:21,100 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4193 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:36:21,100 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3462 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:36:21,100 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2188 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:36:23,476 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5. because 7534b3b1812808d2d3c16342fbf347a5/e has an old edit so flush to free WALs after random delay 42243 ms 2022-03-05 05:36:24,658 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 4333b489291b1f77671b8f2e78779a1e 1/1 column families, dataSize=7.50 KB heapSize=11.98 KB 2022-03-05 05:36:25,584 INFO [MemStoreFlusher.0] retry.RetryInvocationHandler: rg.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 05:36:25,876 INFO [MemStoreFlusher.0] retry.RetryInvocationHandler: at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:614) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.complete over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 05:36:27,300 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 80a1ffe1dc2d8f258a6367d204dc64df 1/1 column families, dataSize=287.55 KB heapSize=536.38 KB 2022-03-05 05:36:27,520 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=7.50 KB at sequenceid=198 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/.tmp/__status_family___/cc43a21e737c4039859495e2f7875c54 2022-03-05 05:36:27,630 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/cc43a21e737c4039859495e2f7875c54, entries=3, sequenceid=198, filesize=5.1 K 2022-03-05 05:36:27,639 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~7.50 KB/7679, heapSize ~11.96 KB/12248, currentSize=0 B/0 for 4333b489291b1f77671b8f2e78779a1e in 2981ms, sequenceid=198, compaction requested=false 2022-03-05 05:36:27,685 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=287.55 KB at sequenceid=3291 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/.tmp/e/8dfc9463da1542c1bff18be9d9a0b5c4 2022-03-05 05:36:27,721 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/8dfc9463da1542c1bff18be9d9a0b5c4, entries=3395, sequenceid=3291, filesize=37.8 K 2022-03-05 05:36:27,725 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~287.55 KB/294452, heapSize ~536.36 KB/549232, currentSize=183 B/183 for 80a1ffe1dc2d8f258a6367d204dc64df in 425ms, sequenceid=3291, compaction requested=true 2022-03-05 05:36:27,725 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 80a1ffe1dc2d8f258a6367d204dc64df/e in aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df. 2022-03-05 05:36:27,726 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/ed2fa91de52e452292e744db7e9cea0f, hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/e6b6a1b3ae30483e860c86f3c706176b, hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/8dfc9463da1542c1bff18be9d9a0b5c4] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/.tmp, totalSize=147.0 K 2022-03-05 05:36:27,808 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 80a1ffe1dc2d8f258a6367d204dc64df#e#compaction#16 average throughput is 25.54 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 05:36:27,939 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 80a1ffe1dc2d8f258a6367d204dc64df/e of 80a1ffe1dc2d8f258a6367d204dc64df into c2aad83face044229391de152709b076(size=141.6 K), total size for store is 141.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 05:36:27,939 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., storeName=80a1ffe1dc2d8f258a6367d204dc64df/e, priority=13, startTime=1646458587725; duration=0sec 2022-03-05 05:36:35,180 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1044 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:36:36,167 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 532 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:36:40,142 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1275 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:36:40,159 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1290 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:36:42,108 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 05:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=306.45 KB, freeSize=306.90 MB, max=307.20 MB, blockCount=15, accesses=4515, hits=4455, hitRatio=98.67%, , cachingAccesses=4471, cachingHits=4454, cachingHitsRatio=99.62%, evictions=735, evicted=0, evictedPerRun=0.0 2022-03-05 05:36:47,763 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1041ms No GCs detected 2022-03-05 05:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 05:36:54,707 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1549 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:36:58,672 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2894ms No GCs detected 2022-03-05 05:37:00,697 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1657 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:37:05,728 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 7534b3b1812808d2d3c16342fbf347a5 1/1 column families, dataSize=170.66 KB heapSize=316.26 KB 2022-03-05 05:37:05,770 INFO [MemStoreFlusher.1] retry.RetryInvocationHandler: rg.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 05:37:06,611 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=170.66 KB at sequenceid=2536 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/.tmp/e/772c759958594202a2f49aabc8e07210 2022-03-05 05:37:06,752 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/772c759958594202a2f49aabc8e07210, entries=1960, sequenceid=2536, filesize=23.6 K 2022-03-05 05:37:06,753 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~170.66 KB/174760, heapSize ~316.24 KB/323832, currentSize=0 B/0 for 7534b3b1812808d2d3c16342fbf347a5 in 1025ms, sequenceid=2536, compaction requested=false 2022-03-05 05:37:08,507 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 516 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:37:22,334 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 9b121c22e18472616debd0996ee2a4cb 1/1 column families, dataSize=577.43 KB heapSize=1.04 MB 2022-03-05 05:37:24,928 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1456ms No GCs detected 2022-03-05 05:37:24,933 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb. because ccdacf4b414cb8d1bc548efc77c663cb/e has an old edit so flush to free WALs after random delay 46313 ms 2022-03-05 05:37:25,440 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=577.43 KB at sequenceid=4157 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/.tmp/e/d0613fde45fa412d83a423091ac9d285 2022-03-05 05:37:25,455 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/d0613fde45fa412d83a423091ac9d285, entries=6703, sequenceid=4157, filesize=68.7 K 2022-03-05 05:37:25,459 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~577.43 KB/591287, heapSize ~1.04 MB/1094160, currentSize=374 B/374 for 9b121c22e18472616debd0996ee2a4cb in 3125ms, sequenceid=4157, compaction requested=true 2022-03-05 05:37:25,460 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 9b121c22e18472616debd0996ee2a4cb/e in aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb. 2022-03-05 05:37:25,460 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/1b6efc90563e4d3c8fe0abed8ad2f6d2, hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/2dc27d135e10443d85b6c02899a10066, hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/d0613fde45fa412d83a423091ac9d285] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/.tmp, totalSize=194.7 K 2022-03-05 05:37:25,592 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 9b121c22e18472616debd0996ee2a4cb#e#compaction#19 average throughput is 29.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 05:37:25,884 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 9b121c22e18472616debd0996ee2a4cb/e of 9b121c22e18472616debd0996ee2a4cb into c3d0c108c9db4da9a93a2dd4e7120638(size=198.1 K), total size for store is 198.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 05:37:25,884 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., storeName=9b121c22e18472616debd0996ee2a4cb/e, priority=13, startTime=1646458645459; duration=0sec 2022-03-05 05:37:42,353 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 629 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:37:42,950 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 968 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:37:44,452 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2014ms No GCs detected 2022-03-05 05:37:56,790 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1291ms No GCs detected 2022-03-05 05:38:11,246 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing ccdacf4b414cb8d1bc548efc77c663cb 1/1 column families, dataSize=282.17 KB heapSize=488.12 KB 2022-03-05 05:38:12,116 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=282.17 KB at sequenceid=1809 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/.tmp/e/2af96cab75134fd28349a30585b1f675 2022-03-05 05:38:12,146 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/2af96cab75134fd28349a30585b1f675, entries=2794, sequenceid=1809, filesize=30 K 2022-03-05 05:38:12,147 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~282.17 KB/288937, heapSize ~488.10 KB/499816, currentSize=0 B/0 for ccdacf4b414cb8d1bc548efc77c663cb in 901ms, sequenceid=1809, compaction requested=true 2022-03-05 05:38:12,148 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of ccdacf4b414cb8d1bc548efc77c663cb/e in aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb. 2022-03-05 05:38:12,148 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/9c9dfd51990f4f16bd55aa8267b030fe, hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/4dc4fc5a78d64a1194c0d9cc47286201, hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/2af96cab75134fd28349a30585b1f675] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/.tmp, totalSize=133.1 K 2022-03-05 05:38:12,210 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: ccdacf4b414cb8d1bc548efc77c663cb#e#compaction#21 average throughput is 46.76 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 05:38:12,356 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in ccdacf4b414cb8d1bc548efc77c663cb/e of ccdacf4b414cb8d1bc548efc77c663cb into 8719de3605884dc2a02d513f519243b3(size=130.6 K), total size for store is 130.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 05:38:12,356 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., storeName=ccdacf4b414cb8d1bc548efc77c663cb/e, priority=13, startTime=1646458692148; duration=0sec 2022-03-05 05:38:20,239 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2256 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:38:33,408 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391. because 0a764e51258807790269e46c55b99391/e has an old edit so flush to free WALs after random delay 223035 ms 2022-03-05 05:38:35,154 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 961 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:38:40,853 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1314ms No GCs detected 2022-03-05 05:38:50,851 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 05:38:53,727 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1461 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:38:53,727 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 569 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:40:02,235 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 753 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:40:02,235 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 736 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:40:14,438 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3206 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:40:14,438 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2539 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:40:14,438 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2424 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:40:14,438 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1367 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:40:23,075 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 780 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:40:48,041 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1470ms No GCs detected 2022-03-05 05:40:50,052 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2005 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:40:53,023 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 05:41:00,753 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2813 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:41:00,778 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2509 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:41:00,778 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1832 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:41:00,778 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1592 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:41:12,674 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 655 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:41:17,532 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1038 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:41:17,559 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1026 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:41:17,559 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 924 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:41:23,361 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1036 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:41:33,338 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 951 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:41:33,338 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 552 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:41:33,338 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 552 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=303.46 KB, freeSize=306.90 MB, max=307.20 MB, blockCount=11, accesses=4582, hits=4482, hitRatio=97.82%, , cachingAccesses=4496, cachingHits=4479, cachingHitsRatio=99.62%, evictions=765, evicted=0, evictedPerRun=0.0 2022-03-05 05:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 05:41:56,592 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 621 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:42:16,444 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 0a764e51258807790269e46c55b99391 1/1 column families, dataSize=955.49 KB heapSize=1.68 MB 2022-03-05 05:42:16,516 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=955.49 KB at sequenceid=5123 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/.tmp/e/e4bfe8dafe3c43d087f15bab3a0b64ed 2022-03-05 05:42:16,531 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/e4bfe8dafe3c43d087f15bab3a0b64ed, entries=10294, sequenceid=5123, filesize=92.6 K 2022-03-05 05:42:16,532 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~955.49 KB/978423, heapSize ~1.68 MB/1759280, currentSize=0 B/0 for 0a764e51258807790269e46c55b99391 in 89ms, sequenceid=5123, compaction requested=true 2022-03-05 05:42:16,533 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 0a764e51258807790269e46c55b99391/e in aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391. 2022-03-05 05:42:16,533 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/fee1de4b85094bf98768706dbe58df4c, hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/5e887b8fd93849e2983cc4ce3833c80d, hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/e4bfe8dafe3c43d087f15bab3a0b64ed] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/.tmp, totalSize=250.9 K 2022-03-05 05:42:16,609 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 0a764e51258807790269e46c55b99391#e#compaction#23 average throughput is 41.14 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 05:42:16,709 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 0a764e51258807790269e46c55b99391/e of 0a764e51258807790269e46c55b99391 into e741166706934b59a37722f54787b6b3(size=261.7 K), total size for store is 261.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 05:42:16,709 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391., storeName=0a764e51258807790269e46c55b99391/e, priority=13, startTime=1646458936532; duration=0sec 2022-03-05 05:42:31,996 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1020 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:42:31,997 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 970 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:42:43,365 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 503 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:42:54,965 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2642 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:42:54,965 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1829 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:42:54,965 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1696 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:44:02,801 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 670 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:44:02,801 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 670 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:44:15,127 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1878 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:44:36,877 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2606ms No GCs detected 2022-03-05 05:44:36,977 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2978 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:44:36,977 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2788 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:44:42,219 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646458473563 with entries=1387, filesize=610.23 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646459081967 2022-03-05 05:44:42,219 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646454683911 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646454683911 2022-03-05 05:44:43,636 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646454969772 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646454969772 2022-03-05 05:44:43,705 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646455573147 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646455573147 2022-03-05 05:44:43,736 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646455611758 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646455611758 2022-03-05 05:44:43,753 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646456204673 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646456204673 2022-03-05 05:44:43,772 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646456801459 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646456801459 2022-03-05 05:44:43,798 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646456835538 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646456835538 2022-03-05 05:44:43,860 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646456862777 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646456862777 2022-03-05 05:44:43,986 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646457273799 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646457273799 2022-03-05 05:44:44,206 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646457338432 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646457338432 2022-03-05 05:44:44,439 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646457944859 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646457944859 2022-03-05 05:44:44,699 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646458161933 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646458161933 2022-03-05 05:44:45,007 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646458209782 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646458209782 2022-03-05 05:44:45,343 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646458419105 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646458419105 2022-03-05 05:44:56,779 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 5784ms No GCs detected 2022-03-05 05:45:01,627 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.sync(AsyncProtobufLogWriter.java:148) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:394) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.consume(AsyncFSWAL.java:581) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:45:04,603 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1325ms No GCs detected 2022-03-05 05:45:15,071 WARN [regionserver/cvp328:16201.logRoller] asyncfs.FanOutOneBlockAsyncDFSOutputHelper: create fan-out dfs output /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646459094248.meta failed, retry = 0 java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$3.userEventTriggered(FanOutOneBlockAsyncDFSOutputHelper.java:386) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:45:15,249 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646458486525.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646459094248.meta 2022-03-05 05:45:15,249 WARN [Close-WAL-Writer-21] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:45:15,250 WARN [Close-WAL-Writer-21] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:45:15,250 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646458486525.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646458486525.meta 2022-03-05 05:45:15,250 INFO [Close-WAL-Writer-21] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646458486525.meta 2022-03-05 05:45:15,277 INFO [Close-WAL-Writer-21] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646458486525.meta after 27ms 2022-03-05 05:45:19,279 WARN [Close-WAL-Writer-21] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646458486525.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646458486525.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 05:45:23,972 WARN [Close-WAL-Writer-16] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:45:23,972 WARN [Close-WAL-Writer-16] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:45:23,972 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646459081967 with entries=14, filesize=6.15 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646459115250 2022-03-05 05:45:23,972 INFO [Close-WAL-Writer-16] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646459081967 2022-03-05 05:45:23,975 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646459101571","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":22403,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:45:23,975 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646459101571","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":22403,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:45:23,975 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646459101772","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":22203,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":202,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:45:23,975 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646459102143","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":21832,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:45:23,976 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646459102279","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":21696,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:45:23,976 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646459102715","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":21261,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":436,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:45:23,976 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646459102289","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":21687,"client":"172.30.41.155:52176","multi.service_calls":0,"queuetimems":8,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:45:23,981 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646459102279","responsesize":"18","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":21702,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-05 05:45:23,976 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646459102279","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":21696,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:45:23,975 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646459102279","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":21696,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:45:23,975 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646459102279","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":21696,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:45:23,975 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646459102279","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":21696,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:45:23,975 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646459102279","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":21696,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:45:23,975 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646459102144","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":21831,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":2,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:45:23,975 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646459102279","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":21696,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:45:23,975 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646459102277","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":21698,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:45:24,000 INFO [Close-WAL-Writer-16] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646459081967 after 28ms 2022-03-05 05:45:28,004 INFO [Close-WAL-Writer-16] util.RecoverLeaseFSUtils: Recovered lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646459081967 after 4032ms 2022-03-05 05:45:30,729 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 979 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:45:30,730 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 980 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:45:32,114 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1257 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:45:32,114 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 932 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:45:32,114 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 932 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:45:45,558 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 565 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:45:45,558 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 558 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:45:46,504 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 827 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:46:30,054 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2991ms No GCs detected 2022-03-05 05:46:30,066 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3478 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:46:30,066 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3400 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:46:30,066 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2252 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:46:30,066 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1728 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:46:30,066 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1721 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:46:32,482 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1502 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:46:44,730 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=4642, hits=4506, hitRatio=97.07%, , cachingAccesses=4518, cachingHits=4501, cachingHitsRatio=99.62%, evictions=794, evicted=0, evictedPerRun=0.0 2022-03-05 05:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 05:46:56,269 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2540ms No GCs detected 2022-03-05 05:46:57,944 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1080 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:47:04,487 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1036 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:47:06,637 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2866ms No GCs detected 2022-03-05 05:47:07,610 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 951 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:47:16,717 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1780ms No GCs detected 2022-03-05 05:47:16,810 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Connection to 172.31.0.219/172.31.0.219:15010 closed at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$channelInactive$2(FanOutOneBlockAsyncDFSOutput.java:286) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.channelInactive(FanOutOneBlockAsyncDFSOutput.java:285) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:389) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:354) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelInactive(IdleStateHandler.java:277) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:831) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:47:16,812 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Connection to 172.31.0.219/172.31.0.219:15010 closed at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$channelInactive$2(FanOutOneBlockAsyncDFSOutput.java:286) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.channelInactive(FanOutOneBlockAsyncDFSOutput.java:285) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:389) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:354) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelInactive(IdleStateHandler.java:277) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:831) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:47:16,812 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Connection to 172.31.0.219/172.31.0.219:15010 closed at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$channelInactive$2(FanOutOneBlockAsyncDFSOutput.java:286) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.channelInactive(FanOutOneBlockAsyncDFSOutput.java:285) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:389) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:354) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelInactive(IdleStateHandler.java:277) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:831) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:47:16,812 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Connection to 172.31.0.219/172.31.0.219:15010 closed at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$channelInactive$2(FanOutOneBlockAsyncDFSOutput.java:286) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.channelInactive(FanOutOneBlockAsyncDFSOutput.java:285) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:389) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:354) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelInactive(IdleStateHandler.java:277) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:831) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:47:16,812 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.sync(AsyncProtobufLogWriter.java:148) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:394) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.consume(AsyncFSWAL.java:581) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:47:19,496 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2223ms No GCs detected 2022-03-05 05:47:21,534 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:568) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.addBlock over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 05:47:21,563 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:568) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.addBlock over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 1 failover attempts. Trying to failover after sleeping for 886ms. 2022-03-05 05:47:22,452 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: lientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:568) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.addBlock over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 2 failover attempts. Trying to failover after sleeping for 1655ms. 2022-03-05 05:47:24,847 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646459115250 with entries=316, filesize=134.06 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646459236812 2022-03-05 05:47:24,847 WARN [Close-WAL-Writer-17] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:47:24,848 WARN [Close-WAL-Writer-17] wal.AsyncProtobufLogWriter: normal close failed, try recover java.lang.IllegalStateException: should call flush first before calling close at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:510) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:514) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:47:24,848 INFO [Close-WAL-Writer-17] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646459115250 2022-03-05 05:47:24,849 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646459228425","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":16424,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:47:24,849 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646459228425","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":16424,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:47:45,776 WARN [regionserver/cvp328:16201] util.Sleeper: We slept 24064ms instead of 5000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired 2022-03-05 05:47:45,798 INFO [Close-WAL-Writer-17] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646459115250 after 20950ms 2022-03-05 05:47:45,795 INFO [regionserver/cvp328:16201.Chore.3] hbase.ScheduledChore: Chore: CompactionChecker missed its start time 2022-03-05 05:47:45,777 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Connection to 172.30.41.155/172.30.41.155:15010 closed at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$channelInactive$2(FanOutOneBlockAsyncDFSOutput.java:286) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.channelInactive(FanOutOneBlockAsyncDFSOutput.java:285) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:389) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:354) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelInactive(IdleStateHandler.java:277) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:831) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:47:45,776 WARN [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 19037ms No GCs detected 2022-03-05 05:47:46,156 INFO [regionserver/cvp328:16201.Chore.1] hbase.ScheduledChore: Chore: MemstoreFlusherChore missed its start time 2022-03-05 05:47:46,257 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: rg.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 05:47:46,257 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 05:47:46,257 WARN [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: A failover has occurred since the start of call #1790 ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 2022-03-05 05:47:50,800 INFO [Close-WAL-Writer-17] retry.RetryInvocationHandler: .apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.recoverLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 05:47:50,801 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 638ms. 2022-03-05 05:47:50,800 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: NamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 708ms. 2022-03-05 05:47:50,829 INFO [Close-WAL-Writer-17] retry.RetryInvocationHandler: menodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.recoverLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 1 failover attempts. Trying to failover after sleeping for 781ms. 2022-03-05 05:47:51,451 WARN [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: A failover has occurred since the start of call #1790 ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 2022-03-05 05:47:51,459 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: tNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 2 failover attempts. Trying to failover after sleeping for 1201ms. 2022-03-05 05:47:51,530 WARN [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: A failover has occurred since the start of call #1791 ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 2022-03-05 05:47:51,532 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: amenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 2 failover attempts. Trying to failover after sleeping for 1651ms. 2022-03-05 05:47:51,754 INFO [Close-WAL-Writer-17] util.RecoverLeaseFSUtils: Recovered lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646459115250 after 26906ms 2022-03-05 05:47:52,660 WARN [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: A failover has occurred since the start of call #1790 ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 2022-03-05 05:47:52,661 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: tNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 3 failover attempts. Trying to failover after sleeping for 2598ms. 2022-03-05 05:47:53,183 WARN [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: A failover has occurred since the start of call #1791 ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 2022-03-05 05:47:53,184 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: amenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 3 failover attempts. Trying to failover after sleeping for 1707ms. 2022-03-05 05:47:54,893 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: amenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 4 failover attempts. Trying to failover after sleeping for 3078ms. 2022-03-05 05:47:55,259 WARN [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: A failover has occurred since the start of call #1790 ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 2022-03-05 05:47:55,262 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: tNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 4 failover attempts. Trying to failover after sleeping for 4084ms. 2022-03-05 05:47:58,075 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646459094248.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646459266151.meta 2022-03-05 05:47:58,075 WARN [Close-WAL-Writer-22] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:47:58,075 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646459094248.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646459094248.meta 2022-03-05 05:47:58,078 WARN [Close-WAL-Writer-22] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:47:58,078 INFO [Close-WAL-Writer-22] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646459094248.meta 2022-03-05 05:47:58,087 INFO [Close-WAL-Writer-22] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646459094248.meta after 9ms 2022-03-05 05:47:58,111 WARN [Close-WAL-Writer-17] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:47:58,111 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646459236812 with entries=26, filesize=14.51 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646459278076 2022-03-05 05:47:58,112 WARN [Close-WAL-Writer-17] wal.AsyncProtobufLogWriter: normal close failed, try recover java.lang.IllegalStateException: should call flush first before calling close at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:510) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:514) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:47:58,112 INFO [Close-WAL-Writer-17] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646459236812 2022-03-05 05:47:58,114 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646459265790","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":12324,"client":"172.30.41.155:52176","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:47:58,114 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646459265794","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":12320,"client":"172.30.41.155:52176","multi.service_calls":0,"queuetimems":4,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:47:58,114 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646459265794","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":12320,"client":"172.30.41.155:52176","multi.service_calls":0,"queuetimems":4,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:47:58,114 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646459265791","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":12323,"client":"172.30.41.155:52176","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:47:58,114 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646459247592","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":30522,"client":"172.30.41.155:52176","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:47:58,114 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646459265792","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":12322,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":2,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:47:58,115 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646459265792","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":12323,"client":"172.30.41.155:52176","multi.service_calls":0,"queuetimems":2,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:47:58,115 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646459265794","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":12321,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":4,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:47:58,115 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646459265794","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":12321,"client":"172.30.41.155:52176","multi.service_calls":0,"queuetimems":4,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:47:58,115 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646459265798","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":12317,"client":"172.30.41.155:52176","multi.service_calls":0,"queuetimems":18206,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:47:58,115 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646459265792","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":12322,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":2,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:47:58,115 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646459265792","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":12322,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":2,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:47:58,115 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646459265790","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":12324,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:47:58,115 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646459265791","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":12323,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:47:58,115 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646459265794","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":12320,"client":"172.30.41.155:52176","multi.service_calls":0,"queuetimems":4,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:47:58,114 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646459265791","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":12323,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:47:58,132 INFO [Close-WAL-Writer-17] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646459236812 after 20ms 2022-03-05 05:47:59,346 WARN [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: A failover has occurred since the start of call #1790 ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 2022-03-05 05:48:02,133 INFO [Close-WAL-Writer-17] util.RecoverLeaseFSUtils: Recovered lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646459236812 after 4021ms 2022-03-05 05:48:02,366 WARN [Close-WAL-Writer-22] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646459094248.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646459094248.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 05:48:14,084 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1015 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:48:26,757 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 744 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:49:01,467 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1850 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:49:01,467 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1848 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:49:01,467 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1771 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:49:01,468 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1036 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:49:01,468 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 668 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:49:11,331 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 761 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:49:56,159 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1067ms No GCs detected 2022-03-05 05:50:19,924 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4262 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:50:19,925 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2515 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:50:19,925 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2426 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:50:19,925 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 986 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:50:31,350 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 527 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:50:54,121 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 881 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:50:54,122 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 841 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:51:28,426 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 951 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:51:28,426 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 749 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:51:28,426 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 740 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:51:28,426 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 515 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:51:34,925 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 625 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:51:43,186 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1438 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:51:43,194 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 697 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=4652, hits=4516, hitRatio=97.08%, , cachingAccesses=4528, cachingHits=4511, cachingHitsRatio=99.62%, evictions=822, evicted=0, evictedPerRun=0.0 2022-03-05 05:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 05:51:58,933 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 797 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:52:04,617 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 736 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:52:13,999 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2712 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:52:13,999 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1254 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:52:13,999 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1237 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:52:13,999 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1084 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:52:27,114 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1924 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:52:27,128 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1875 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:52:27,128 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1038 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:52:27,128 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 880 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:53:23,687 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1002 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:53:23,831 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 747 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:53:23,831 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 535 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:53:56,103 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 10075 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:53:56,103 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Requesting log roll because we exceeded slow sync threshold; time=10075 ms, threshold=10000 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:53:56,103 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 9752 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:53:56,103 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 8554 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:53:56,103 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 7595 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 05:53:56,105 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646459626023","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x96\\xC4\\x07Devi ","processingtimems":10081,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 05:53:56,622 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646459278076 with entries=834, filesize=367.96 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646459636103 2022-03-05 05:54:09,891 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 620 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:54:12,648 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 636 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:54:36,447 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 632 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:55:13,817 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 966 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:55:42,184 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1544 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:55:52,074 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1011ms No GCs detected 2022-03-05 05:55:56,524 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 511 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:55:56,524 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 508 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:56:04,835 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1008ms No GCs detected 2022-03-05 05:56:07,591 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2755 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:56:07,591 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2689 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:56:07,591 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2685 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:56:07,591 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2680 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:56:07,591 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2680 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:56:07,591 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1804 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:56:07,591 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 761 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:56:07,591 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 761 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:56:09,772 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1963 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:56:09,772 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 774 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:56:10,691 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 923 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:56:14,569 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3226 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:56:15,189 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3023 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:56:15,190 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1936 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:56:15,190 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1915 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:56:27,505 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1234 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:56:27,505 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 745 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:56:33,308 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1135 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:56:40,733 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 642 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=301.29 KB, freeSize=306.91 MB, max=307.20 MB, blockCount=9, accesses=4683, hits=4543, hitRatio=97.01%, , cachingAccesses=4559, cachingHits=4538, cachingHitsRatio=99.54%, evictions=852, evicted=0, evictedPerRun=0.0 2022-03-05 05:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 05:56:46,724 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4974 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:56:46,721 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1514ms No GCs detected 2022-03-05 05:56:51,629 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 9706 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:56:51,629 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 9682 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:56:51,629 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 9649 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:56:51,629 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 8775 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:56:51,753 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1281 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:56:51,807 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 05:56:53,435 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1661 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:56:54,011 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1670 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:56:54,293 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1077 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:56:54,293 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1069 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:56:54,299 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 863 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:56:56,616 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 564 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:56:57,894 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 943 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:56:59,557 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 897 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:56:59,560 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 900 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:57:00,256 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 589 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:57:00,256 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 589 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:57:00,648 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646459266151.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646459819539.meta 2022-03-05 05:57:00,649 WARN [Close-WAL-Writer-23] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:57:00,649 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646459266151.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646459266151.meta 2022-03-05 05:57:00,649 WARN [Close-WAL-Writer-23] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 05:57:00,649 INFO [Close-WAL-Writer-23] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646459266151.meta 2022-03-05 05:57:00,664 INFO [Close-WAL-Writer-23] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646459266151.meta after 15ms 2022-03-05 05:57:03,439 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1284 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:57:03,439 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1283 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:57:03,439 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 681 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:57:04,666 WARN [Close-WAL-Writer-23] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646459266151.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646459266151.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 05:57:12,901 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 707 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:57:12,901 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 697 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:57:46,275 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3359 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:57:46,275 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1371 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:57:46,275 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1288 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:57:52,794 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 549 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:58:15,535 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 611 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:58:15,535 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 532 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:59:18,177 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 614 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:59:19,001 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 823 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:59:37,540 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1337 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:59:39,391 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1036 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:59:39,391 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 972 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:59:46,939 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1206 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 05:59:59,228 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1090 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:00:13,989 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1207ms No GCs detected 2022-03-05 06:00:13,992 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1792 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:00:27,094 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 916 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:00:27,094 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 873 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:00:46,327 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 653 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:00:49,372 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 738 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:00:55,907 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 942 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:00:55,954 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 973 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:01:00,387 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 06:01:00,393 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 838 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:01:00,993 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 612 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:01:01,634 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1250 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:01:01,634 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 641 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:01:08,771 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1010 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:01:08,988 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 793 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:01:14,716 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2329ms No GCs detected 2022-03-05 06:01:17,685 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 8445 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:01:17,685 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 7127 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:01:17,685 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6385 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:01:17,685 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5985 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:01:17,685 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5656 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:01:17,685 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2956 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:01:17,685 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2758 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:01:17,685 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2613 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:01:17,685 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2033 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:01:17,685 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1356 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:01:23,490 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1179 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:01:31,789 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2027 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:01:31,790 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2026 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:01:31,790 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1001 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:01:31,790 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 998 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:01:33,486 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1210ms No GCs detected 2022-03-05 06:01:37,928 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6138 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:01:40,497 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2568 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:01:40,497 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1901 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:01:40,497 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1044 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:01:40,772 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:568) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.addBlock over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 06:01:40,789 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:568) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.addBlock over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 1 failover attempts. Trying to failover after sleeping for 318ms. 2022-03-05 06:01:41,109 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: lientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:568) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.addBlock over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 2 failover attempts. Trying to failover after sleeping for 1111ms. 2022-03-05 06:01:44,943 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=301.29 KB, freeSize=306.91 MB, max=307.20 MB, blockCount=9, accesses=4704, hits=4564, hitRatio=97.02%, , cachingAccesses=4580, cachingHits=4559, cachingHitsRatio=99.54%, evictions=882, evicted=0, evictedPerRun=0.0 2022-03-05 06:01:46,336 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3431 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:01:46,337 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1392 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:01:46,337 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1267 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:01:46,499 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646459819539.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646460097925.meta 2022-03-05 06:01:46,500 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646459819539.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646459819539.meta 2022-03-05 06:01:46,504 WARN [Close-WAL-Writer-24] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:01:46,504 WARN [Close-WAL-Writer-24] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:01:46,505 INFO [Close-WAL-Writer-24] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646459819539.meta 2022-03-05 06:01:46,556 INFO [Close-WAL-Writer-24] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646459819539.meta after 51ms 2022-03-05 06:01:48,028 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 06:01:50,558 WARN [Close-WAL-Writer-24] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646459819539.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646459819539.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 06:01:59,333 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1953 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:01:59,333 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1604 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:01:59,333 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1391 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:01:59,333 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1191 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:01:59,333 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 714 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:01:59,333 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 714 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:02:09,112 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1245 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:02:28,525 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1888ms No GCs detected 2022-03-05 06:02:34,025 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 7338 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:02:34,026 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2387ms No GCs detected 2022-03-05 06:02:34,137 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5619 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:02:34,137 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5597 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:02:34,137 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4979 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:02:34,137 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4943 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:02:34,142 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2688 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:02:34,142 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2687 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:02:34,142 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2649 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:02:34,142 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1660 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:02:34,142 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 657 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:02:44,096 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3398 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:02:44,096 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2471 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:02:44,223 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2486 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:02:44,223 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2040 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:02:44,223 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 858 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:03:28,754 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 667 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:03:29,736 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1393 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:03:29,736 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1056 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:03:29,736 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1056 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:03:29,736 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 982 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:03:56,531 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646459636103 with entries=1298, filesize=569.54 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646460236510 2022-03-05 06:05:01,264 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 874 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:05:01,264 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 630 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:05:12,746 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 989 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:05:12,746 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 778 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:05:41,809 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 583 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:05:48,465 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1015 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:06:15,969 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 06:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=301.29 KB, freeSize=306.91 MB, max=307.20 MB, blockCount=9, accesses=4729, hits=4589, hitRatio=97.04%, , cachingAccesses=4605, cachingHits=4584, cachingHitsRatio=99.54%, evictions=912, evicted=0, evictedPerRun=0.0 2022-03-05 06:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 06:08:23,303 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 853 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:08:29,297 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1214 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:08:29,297 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1215 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:08:29,299 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 668 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:08:29,299 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 668 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:09:14,034 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2023 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:09:14,034 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1120 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:09:37,278 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1530 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:09:47,591 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 06:09:59,883 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3837 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:09:59,884 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3248 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:09:59,884 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2913 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:09:59,884 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1267 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:09:59,884 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1258 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=301.29 KB, freeSize=306.91 MB, max=307.20 MB, blockCount=9, accesses=4751, hits=4611, hitRatio=97.05%, , cachingAccesses=4627, cachingHits=4606, cachingHitsRatio=99.55%, evictions=942, evicted=0, evictedPerRun=0.0 2022-03-05 06:11:47,424 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646460097925.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646460707410.meta 2022-03-05 06:11:47,424 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646460097925.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646460097925.meta 2022-03-05 06:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 06:12:12,506 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 722 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:12:16,099 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1058 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:12:34,290 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1259 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:12:34,290 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 712 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:12:34,290 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 711 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:12:34,290 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 711 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:12:34,290 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 711 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:12:34,290 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 711 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:12:35,326 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1253ms No GCs detected 2022-03-05 06:12:39,070 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1001ms No GCs detected 2022-03-05 06:12:39,078 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3394 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:12:40,650 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4865 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:12:40,650 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4692 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:12:42,747 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2096 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:12:42,747 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1044 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:12:42,747 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 948 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:12:42,770 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 761 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:12:43,421 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 556 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:12:52,742 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1270ms No GCs detected 2022-03-05 06:12:58,160 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:13:05,710 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:13:05,710 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:13:05,710 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:13:05,710 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:13:05,797 WARN [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 12554ms No GCs detected 2022-03-05 06:13:18,048 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1973ms No GCs detected 2022-03-05 06:13:18,634 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: 803) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:469) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.create over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 06:13:18,655 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: lPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:469) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.create over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 787ms. 2022-03-05 06:13:19,541 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: PB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:469) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.create over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 2 failover attempts. Trying to failover after sleeping for 1188ms. 2022-03-05 06:13:25,977 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646460236510 with entries=1165, filesize=504.58 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646460785715 2022-03-05 06:13:25,978 WARN [Close-WAL-Writer-20] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:13:25,980 WARN [Close-WAL-Writer-20] wal.AsyncProtobufLogWriter: normal close failed, try recover java.lang.IllegalStateException: should call flush first before calling close at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:510) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:514) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:13:25,980 INFO [Close-WAL-Writer-20] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646460236510 2022-03-05 06:13:25,994 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646460772741","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":33253,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":7,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:13:25,994 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646460774897","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":31097,"client":"172.30.41.155:52176","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:13:25,995 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646460776424","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":29570,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":5,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:13:25,998 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646460776634","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":29364,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:13:25,995 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646460776435","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":29559,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:13:25,998 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646460781512","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":24486,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:13:25,998 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646460776435","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":29563,"client":"172.30.41.155:52176","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:13:26,011 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646460786965","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":19046,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":3983,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:13:26,011 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646460786280","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":19731,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":7333,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:13:26,010 INFO [Close-WAL-Writer-20] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646460236510 after 30ms 2022-03-05 06:13:26,009 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646460778948","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":27061,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:13:26,009 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646460776430","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":29579,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:13:26,009 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646460782886","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":23123,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":4,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:13:26,043 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646460786965","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":19078,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":1292,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:13:26,042 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646460785673","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":20369,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:13:26,042 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646460785673","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":20369,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:13:26,042 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646460785673","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":20369,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:13:26,160 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646460707410.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646460805977.meta 2022-03-05 06:13:26,167 WARN [Close-WAL-Writer-26] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:13:26,167 WARN [Close-WAL-Writer-26] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:13:26,167 INFO [Close-WAL-Writer-26] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646460707410.meta 2022-03-05 06:13:26,171 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646460707410.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646460707410.meta 2022-03-05 06:13:26,215 INFO [Close-WAL-Writer-26] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646460707410.meta after 48ms 2022-03-05 06:13:30,064 INFO [Close-WAL-Writer-20] util.RecoverLeaseFSUtils: Recovered lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646460236510 after 4084ms 2022-03-05 06:13:30,218 WARN [Close-WAL-Writer-26] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646460707410.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646460707410.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 06:13:32,095 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 788 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:13:32,095 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 701 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:13:36,274 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 704 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:13:50,358 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3980ms No GCs detected 2022-03-05 06:13:52,698 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2340 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:13:59,138 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646460805977.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646460836172.meta 2022-03-05 06:13:59,138 WARN [Close-WAL-Writer-26] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:13:59,138 WARN [Close-WAL-Writer-26] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:13:59,139 INFO [Close-WAL-Writer-26] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646460805977.meta 2022-03-05 06:13:59,150 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646460805977.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646460805977.meta 2022-03-05 06:13:59,164 INFO [Close-WAL-Writer-26] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646460805977.meta after 25ms 2022-03-05 06:14:03,165 WARN [Close-WAL-Writer-26] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646460805977.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646460805977.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 06:14:11,991 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 740 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:14:17,966 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 517 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:14:33,246 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1892 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:14:33,246 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1892 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:14:33,258 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1049 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:14:33,258 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1035 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:14:33,258 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 558 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:14:33,258 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 558 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:14:33,258 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 558 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:14:33,258 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 558 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:14:33,258 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 557 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:14:40,369 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1118 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:14:57,688 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 800 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:15:04,591 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1254 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=301.29 KB, freeSize=306.91 MB, max=307.20 MB, blockCount=9, accesses=4773, hits=4633, hitRatio=97.07%, , cachingAccesses=4649, cachingHits=4628, cachingHitsRatio=99.55%, evictions=971, evicted=0, evictedPerRun=0.0 2022-03-05 06:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 06:20:32,292 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1577 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:20:34,176 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3422 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=301.29 KB, freeSize=306.91 MB, max=307.20 MB, blockCount=9, accesses=4795, hits=4655, hitRatio=97.08%, , cachingAccesses=4671, cachingHits=4650, cachingHitsRatio=99.55%, evictions=1001, evicted=0, evictedPerRun=0.0 2022-03-05 06:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 06:22:59,633 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 773 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:22:59,633 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 773 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:23:01,401 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1761 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:23:01,401 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1673 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:23:03,159 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1421ms No GCs detected 2022-03-05 06:23:08,229 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 7108 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:23:08,229 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6947 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:23:08,229 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6828 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:23:08,229 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6794 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:23:08,229 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6295 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:23:08,229 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5070 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:23:08,229 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4958 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:23:08,229 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4893 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:23:08,230 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3984 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:23:08,230 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1544 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:23:09,441 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: rg.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 06:23:09,949 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:568) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.addBlock over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 06:23:09,950 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:568) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.addBlock over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 768ms. 2022-03-05 06:23:10,803 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: lientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:568) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.addBlock over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 2 failover attempts. Trying to failover after sleeping for 1678ms. 2022-03-05 06:23:12,491 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646460785715 with entries=1355, filesize=588.20 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646461389438 2022-03-05 06:24:03,596 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646460836172.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646461442645.meta 2022-03-05 06:24:03,596 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646460836172.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646460836172.meta 2022-03-05 06:24:07,922 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2312 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:24:13,420 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5504 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:24:13,768 WARN [Close-WAL-Writer-27] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:24:15,654 INFO [Close-WAL-Writer-27] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646460836172.meta 2022-03-05 06:24:15,674 WARN [Close-WAL-Writer-27] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646460836172.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646460836172.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 06:24:17,401 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3980 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:24:17,401 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2240 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:24:17,401 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2239 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:24:17,642 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1396 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:24:19,926 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1915 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:24:19,926 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1834 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:24:23,758 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 06:24:27,760 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2847 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:24:41,882 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 16925 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:24:41,882 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Requesting log roll because we exceeded slow sync threshold; time=16925 ms, threshold=10000 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:24:41,883 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.sync(AsyncProtobufLogWriter.java:148) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:394) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.consume(AsyncFSWAL.java:581) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:24:41,883 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646461464957","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":16926,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:24:34,257 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 7147ms No GCs detected 2022-03-05 06:24:44,425 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646461389438 with entries=159, filesize=72.08 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646461481883 2022-03-05 06:24:44,425 WARN [Close-WAL-Writer-22] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:24:44,425 WARN [Close-WAL-Writer-22] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:24:44,426 INFO [Close-WAL-Writer-22] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646461389438 2022-03-05 06:24:44,430 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646461466609","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":17821,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:24:44,430 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646461467897","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":16533,"client":"172.30.41.155:52176","multi.service_calls":0,"queuetimems":149,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:24:44,430 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646461467377","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":17053,"client":"172.30.41.155:52176","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:24:44,432 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646461469101","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":15331,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:24:44,432 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646461470760","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":13672,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:24:44,431 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646461468991","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":15440,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":2016,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:24:44,431 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646461468631","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":15800,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:24:44,434 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646461469101","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":15330,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:24:44,431 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646461466608","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":17823,"client":"172.30.41.155:52176","multi.service_calls":0,"queuetimems":472,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:24:44,431 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646461467919","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":16512,"client":"172.30.41.155:52176","multi.service_calls":0,"queuetimems":171,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:24:44,431 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646461468630","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":15801,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:24:44,450 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646461470202","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":14248,"client":"172.30.41.155:52176","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:24:44,432 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646461470865","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":13567,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:24:44,458 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646461469724","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":14734,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:24:44,476 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646461469556","responsesize":"26","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":14920,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-05 06:24:44,646 INFO [Close-WAL-Writer-22] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646461389438 after 220ms 2022-03-05 06:24:48,771 INFO [Close-WAL-Writer-22] retry.RetryInvocationHandler: .apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.recoverLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 06:24:59,179 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 8909ms No GCs detected 2022-03-05 06:25:06,490 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646461488033","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":18457,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":70,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:25:06,489 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 18456 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:25:10,410 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Requesting log roll because we exceeded slow sync threshold; time=18456 ms, threshold=10000 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:25:10,410 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.sync(AsyncProtobufLogWriter.java:148) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:394) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.consume(AsyncFSWAL.java:581) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:25:13,695 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1209ms No GCs detected 2022-03-05 06:25:20,090 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3611ms No GCs detected 2022-03-05 06:25:22,822 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2232ms No GCs detected 2022-03-05 06:25:30,100 INFO [Close-WAL-Writer-22] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646461389438 after 45674ms 2022-03-05 06:25:30,137 WARN [Close-WAL-Writer-22] util.RecoverLeaseFSUtils: Cannot recoverLease after trying for 23000ms (hbase.lease.recovery.timeout); continuing, but may be DATALOSS!!!; attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646461389438 after 45711ms 2022-03-05 06:25:39,819 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: java.io.EOFException: End of File Exception between local host is: "cvp328.sjc.aristanetworks.com/172.30.41.118"; destination host is: "cvp328.sjc.aristanetworks.com":9001; : java.io.EOFException; For more details see: http://wiki.apache.org/hadoop/EOFException, while invoking ClientNamenodeProtocolTranslatorPB.create over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 06:25:41,378 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646461442645.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646461509179.meta 2022-03-05 06:25:42,624 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646461442645.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646461442645.meta 2022-03-05 06:25:41,378 WARN [Close-WAL-Writer-28] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:25:42,624 WARN [Close-WAL-Writer-28] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:25:42,625 INFO [Close-WAL-Writer-28] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646461442645.meta 2022-03-05 06:25:42,668 INFO [Close-WAL-Writer-28] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646461442645.meta after 43ms 2022-03-05 06:25:42,801 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646461481883 with entries=66, filesize=29.71 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646461542624 2022-03-05 06:25:42,801 WARN [Close-WAL-Writer-22] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:25:42,801 WARN [Close-WAL-Writer-22] wal.AsyncProtobufLogWriter: normal close failed, try recover java.lang.IllegalStateException: should call flush first before calling close at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:510) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:514) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:25:42,801 INFO [Close-WAL-Writer-22] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646461481883 2022-03-05 06:25:42,810 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646461508767","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":34043,"client":"172.30.41.155:52176","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:25:42,813 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646461488764","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":54048,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:25:42,814 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646461506489","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":36325,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":7309,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:25:42,815 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646461499180","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":43635,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:25:42,835 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646461508246","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":34589,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":1757,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:25:42,835 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646461508838","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":33997,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":2349,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:25:42,828 INFO [Close-WAL-Writer-22] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646461481883 after 27ms 2022-03-05 06:25:42,819 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646461506489","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":36330,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:25:42,816 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646461508814","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":34002,"client":"172.30.41.155:52176","multi.service_calls":0,"queuetimems":47,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:25:42,816 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646461508767","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":34049,"client":"172.30.41.155:52176","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:25:42,815 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646461499190","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":43625,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":10,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:25:42,815 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646461499190","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":43625,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":10,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:25:42,815 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646461499180","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":43635,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:25:42,815 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646461499180","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":43635,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:25:42,815 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646461499179","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":43634,"client":"172.30.41.155:52176","multi.service_calls":0,"queuetimems":4761,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:25:42,814 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646461510409","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":32405,"client":"172.30.41.155:52176","multi.service_calls":0,"queuetimems":1642,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:25:42,846 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646461508767","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":34079,"client":"172.30.41.155:52176","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:25:46,670 WARN [Close-WAL-Writer-28] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646461442645.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646461442645.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 06:25:47,774 INFO [Close-WAL-Writer-22] util.RecoverLeaseFSUtils: Recovered lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646461481883 after 4973ms 2022-03-05 06:25:48,645 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 764 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:25:48,645 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 668 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:26:23,240 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1347ms No GCs detected 2022-03-05 06:26:24,916 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1671 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:26:26,379 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2638ms No GCs detected 2022-03-05 06:26:27,010 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 642 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:26:41,828 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 542 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=301.29 KB, freeSize=306.91 MB, max=307.20 MB, blockCount=9, accesses=4871, hits=4731, hitRatio=97.13%, , cachingAccesses=4747, cachingHits=4726, cachingHitsRatio=99.56%, evictions=1029, evicted=0, evictedPerRun=0.0 2022-03-05 06:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 06:26:57,350 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 884 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:26:57,350 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 789 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:28:03,915 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 882 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:28:04,584 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1345 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:28:04,584 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1345 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:28:05,231 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 646 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:28:18,358 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1803 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:28:19,090 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1702 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:28:19,090 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 732 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:28:31,656 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 976 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:28:31,656 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 927 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:28:50,170 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2205ms No GCs detected 2022-03-05 06:28:50,178 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4671 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:28:50,178 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4654 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:28:50,178 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4213 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:28:50,178 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3935 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:28:50,180 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2194 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:28:56,706 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6533 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:29:03,471 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:29:03,550 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:29:03,551 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:29:03,551 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:29:03,551 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:29:03,551 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:29:03,551 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.sync(AsyncProtobufLogWriter.java:148) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:394) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.consume(AsyncFSWAL.java:581) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:29:03,856 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646461542624 with entries=529, filesize=232.37 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646461743554 2022-03-05 06:29:03,856 WARN [Close-WAL-Writer-23] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:29:03,857 WARN [Close-WAL-Writer-23] wal.AsyncProtobufLogWriter: normal close failed, try recover java.lang.IllegalStateException: should call flush first before calling close at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:510) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:514) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:29:03,857 INFO [Close-WAL-Writer-23] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646461542624 2022-03-05 06:29:03,869 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646461733748","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":10121,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:29:03,869 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646461732232","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":11637,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:29:03,887 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646461725964","responsesize":"78","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 11 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x06Kernel ","processingtimems":17923,"client":"172.30.41.155:52234","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":11} 2022-03-05 06:29:04,073 INFO [Close-WAL-Writer-23] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646461542624 after 216ms 2022-03-05 06:29:08,294 INFO [Close-WAL-Writer-23] retry.RetryInvocationHandler: .apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.recoverLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 06:29:10,888 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1241ms No GCs detected 2022-03-05 06:29:17,256 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:29:17,258 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:29:17,258 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:29:17,258 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:29:17,258 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.sync(AsyncProtobufLogWriter.java:148) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:394) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.consume(AsyncFSWAL.java:581) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:29:19,568 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1811ms No GCs detected 2022-03-05 06:29:21,027 INFO [Close-WAL-Writer-23] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646461542624 after 17170ms 2022-03-05 06:29:21,222 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:568) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.addBlock over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 06:29:21,228 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:568) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.addBlock over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 1 failover attempts. Trying to failover after sleeping for 687ms. 2022-03-05 06:29:21,917 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: lientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:568) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.addBlock over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 2 failover attempts. Trying to failover after sleeping for 1569ms. 2022-03-05 06:29:22,031 INFO [Close-WAL-Writer-23] retry.RetryInvocationHandler: apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.isFileClosed(ClientNamenodeProtocolServerSideTranslatorPB.java:1322) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.isFileClosed over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 06:29:23,486 WARN [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: A failover has occurred since the start of call #1963 ClientNamenodeProtocolTranslatorPB.addBlock over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 2022-03-05 06:29:26,910 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646461509179.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646461759654.meta 2022-03-05 06:29:26,910 WARN [Close-WAL-Writer-29] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:29:26,910 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646461509179.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646461509179.meta 2022-03-05 06:29:26,911 WARN [Close-WAL-Writer-29] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:29:26,911 INFO [Close-WAL-Writer-29] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646461509179.meta 2022-03-05 06:29:26,963 INFO [Close-WAL-Writer-29] retry.RetryInvocationHandler: .apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.recoverLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 06:29:26,963 INFO [Log-Archiver-0] retry.RetryInvocationHandler: at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setTimes(ClientNamenodeProtocolServerSideTranslatorPB.java:1062) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.setTimes over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 06:29:26,963 WARN [Log-Archiver-0] retry.RetryInvocationHandler: A failover has occurred since the start of call #1966 ClientNamenodeProtocolTranslatorPB.setTimes over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 2022-03-05 06:29:26,963 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: rg.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 06:29:26,964 WARN [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: A failover has occurred since the start of call #1965 ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 2022-03-05 06:29:26,965 INFO [Close-WAL-Writer-29] retry.RetryInvocationHandler: menodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.recoverLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 573ms. 2022-03-05 06:29:26,967 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: NamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 843ms. 2022-03-05 06:29:26,969 INFO [Log-Archiver-0] retry.RetryInvocationHandler: lientNamenodeProtocolServerSideTranslatorPB.setTimes(ClientNamenodeProtocolServerSideTranslatorPB.java:1062) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.setTimes over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 591ms. 2022-03-05 06:29:27,540 INFO [Close-WAL-Writer-29] retry.RetryInvocationHandler: enodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.recoverLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 2 failover attempts. Trying to failover after sleeping for 1752ms. 2022-03-05 06:29:27,561 WARN [Log-Archiver-0] retry.RetryInvocationHandler: A failover has occurred since the start of call #1966 ClientNamenodeProtocolTranslatorPB.setTimes over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 2022-03-05 06:29:27,563 INFO [Log-Archiver-0] retry.RetryInvocationHandler: lientNamenodeProtocolServerSideTranslatorPB.setTimes(ClientNamenodeProtocolServerSideTranslatorPB.java:1062) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.setTimes over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 2 failover attempts. Trying to failover after sleeping for 603ms. 2022-03-05 06:29:27,810 WARN [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: A failover has occurred since the start of call #1965 ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 2022-03-05 06:29:27,812 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: amenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 2 failover attempts. Trying to failover after sleeping for 1700ms. 2022-03-05 06:29:27,835 INFO [Close-WAL-Writer-23] retry.RetryInvocationHandler: apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.isFileClosed(ClientNamenodeProtocolServerSideTranslatorPB.java:1322) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.isFileClosed over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 06:29:28,166 WARN [Log-Archiver-0] retry.RetryInvocationHandler: A failover has occurred since the start of call #1966 ClientNamenodeProtocolTranslatorPB.setTimes over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 2022-03-05 06:29:29,293 WARN [Close-WAL-Writer-29] retry.RetryInvocationHandler: A failover has occurred since the start of call #1967 ClientNamenodeProtocolTranslatorPB.recoverLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 2022-03-05 06:29:29,295 WARN [Close-WAL-Writer-29] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646461509179.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646461509179.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 06:29:29,512 WARN [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: A failover has occurred since the start of call #1965 ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 2022-03-05 06:29:29,884 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646461743554 with entries=47, filesize=17.23 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646461766910 2022-03-05 06:29:29,884 WARN [Close-WAL-Writer-24] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:29:29,884 WARN [Close-WAL-Writer-24] wal.AsyncProtobufLogWriter: normal close failed, try recover java.lang.IllegalStateException: should call flush first before calling close at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:510) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:514) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:29:29,885 INFO [Close-WAL-Writer-24] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646461743554 2022-03-05 06:29:29,885 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646461747806","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":22079,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:29:29,885 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646461759653","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":10232,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":2402,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:29:29,885 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646461748215","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":21670,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:29:29,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646461749177","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":20708,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:29:29,885 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646461752015","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":17870,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:29:29,886 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646461759654","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":10232,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":2403,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:29:29,886 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646461750631","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":19254,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:29:29,886 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646461757993","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":11893,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:29:29,886 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646461759577","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":10309,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":6785,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:29:30,034 INFO [Close-WAL-Writer-24] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646461743554 after 149ms 2022-03-05 06:29:33,067 INFO [Close-WAL-Writer-23] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=2 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646461542624 after 29210ms 2022-03-05 06:29:33,067 WARN [Close-WAL-Writer-23] util.RecoverLeaseFSUtils: Cannot recoverLease after trying for 23000ms (hbase.lease.recovery.timeout); continuing, but may be DATALOSS!!!; attempt=2 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646461542624 after 29210ms 2022-03-05 06:29:34,035 INFO [Close-WAL-Writer-24] util.RecoverLeaseFSUtils: Recovered lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646461743554 after 4150ms 2022-03-05 06:30:01,363 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1125 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:30:01,363 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 563 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=301.29 KB, freeSize=306.91 MB, max=307.20 MB, blockCount=9, accesses=4893, hits=4753, hitRatio=97.14%, , cachingAccesses=4769, cachingHits=4748, cachingHitsRatio=99.56%, evictions=1058, evicted=0, evictedPerRun=0.0 2022-03-05 06:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-05 06:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 06:31:54,902 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1557 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:31:54,902 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1222 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:31:57,550 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2621 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:31:57,550 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1908 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:31:57,550 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1543 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:31:57,550 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1346 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:31:57,550 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 846 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:32:00,706 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2616 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:32:00,706 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2035 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:32:00,706 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2035 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:32:01,329 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1710 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:32:01,329 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1710 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:32:01,329 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1710 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:32:01,329 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1316 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:32:01,329 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 889 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:32:01,329 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 606 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:32:30,402 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5487 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:32:30,402 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5451 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:32:30,402 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4236 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:32:30,402 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3872 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:32:30,402 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3783 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:32:30,403 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2994 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:32:30,403 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1750 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:32:30,403 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1750 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:32:30,403 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 780 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:32:31,913 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 575 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:32:42,993 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1262 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:32:51,815 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1456ms No GCs detected 2022-03-05 06:33:06,988 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3704ms No GCs detected 2022-03-05 06:33:07,051 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:33:07,110 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:33:07,172 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:33:07,173 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:33:07,173 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:33:07,173 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:33:07,175 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:33:08,541 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646461766910 with entries=495, filesize=221.84 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646461987115 2022-03-05 06:33:08,542 WARN [Close-WAL-Writer-25] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:33:08,542 WARN [Close-WAL-Writer-25] wal.AsyncProtobufLogWriter: normal close failed, try recover java.lang.IllegalStateException: should call flush first before calling close at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:510) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:514) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:33:08,542 INFO [Close-WAL-Writer-25] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646461766910 2022-03-05 06:33:08,655 INFO [Close-WAL-Writer-25] retry.RetryInvocationHandler: .apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.recoverLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 06:33:08,881 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646461978221","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":10660,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":2,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:33:09,656 INFO [Close-WAL-Writer-25] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.recoverLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 1 failover attempts. Trying to failover after sleeping for 363ms. 2022-03-05 06:33:10,094 INFO [Close-WAL-Writer-25] retry.RetryInvocationHandler: enodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.recoverLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 2 failover attempts. Trying to failover after sleeping for 1138ms. 2022-03-05 06:33:13,676 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1143ms No GCs detected 2022-03-05 06:33:15,635 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1857 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:33:18,703 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 722 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:33:20,992 INFO [Close-WAL-Writer-25] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646461766910 after 12450ms 2022-03-05 06:33:26,160 INFO [Close-WAL-Writer-25] retry.RetryInvocationHandler: .apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.recoverLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 06:33:26,171 INFO [Close-WAL-Writer-25] retry.RetryInvocationHandler: menodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.recoverLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 583ms. 2022-03-05 06:33:26,768 INFO [Close-WAL-Writer-25] retry.RetryInvocationHandler: enodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.recoverLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 2 failover attempts. Trying to failover after sleeping for 1415ms. 2022-03-05 06:33:28,194 INFO [Close-WAL-Writer-25] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646461766910 after 19652ms 2022-03-05 06:33:46,800 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 828 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:33:46,801 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 528 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:34:01,289 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 620 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:34:01,289 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 503 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:34:04,726 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2272 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:34:04,726 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2272 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:34:05,359 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1720 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:34:05,359 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1355 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:34:05,359 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1269 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:34:05,359 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1168 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:34:06,776 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1417 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:34:06,781 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1143 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:34:06,783 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] regionserver.MultiVersionConcurrencyControl: STUCK for : 1000 millis. MultiVersionConcurrencyControl{readPoint=4392, writePoint=4394, regionName=80a1ffe1dc2d8f258a6367d204dc64df} 2022-03-05 06:34:10,835 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1641 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:35:32,152 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1347 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:35:32,152 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1170 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:35:32,152 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 976 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:35:32,152 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 920 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:35:32,152 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 796 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:35:41,183 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 642 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:35:42,095 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 690 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:35:45,630 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3039ms No GCs detected 2022-03-05 06:35:46,197 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. because 7d1b53259f1d8280a47f7704f98f2c9f/e has an old edit so flush to free WALs after random delay 269375 ms 2022-03-05 06:35:57,450 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1313 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:36:30,724 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 520 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:36:35,832 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df. because 80a1ffe1dc2d8f258a6367d204dc64df/e has an old edit so flush to free WALs after random delay 11231 ms 2022-03-05 06:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=301.29 KB, freeSize=306.91 MB, max=307.20 MB, blockCount=9, accesses=4921, hits=4781, hitRatio=97.16%, , cachingAccesses=4797, cachingHits=4776, cachingHitsRatio=99.56%, evictions=1088, evicted=0, evictedPerRun=0.0 2022-03-05 06:36:47,063 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 80a1ffe1dc2d8f258a6367d204dc64df 1/1 column families, dataSize=286.46 KB heapSize=534.66 KB 2022-03-05 06:36:47,127 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=286.46 KB at sequenceid=4443 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/.tmp/e/d7dc25ebda75464d83d83c334bf4a5cb 2022-03-05 06:36:47,144 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/d7dc25ebda75464d83d83c334bf4a5cb, entries=3385, sequenceid=4443, filesize=37.2 K 2022-03-05 06:36:47,145 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~286.46 KB/293340, heapSize ~534.64 KB/547472, currentSize=0 B/0 for 80a1ffe1dc2d8f258a6367d204dc64df in 82ms, sequenceid=4443, compaction requested=false 2022-03-05 06:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 06:37:05,798 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e. because 4333b489291b1f77671b8f2e78779a1e/__status_family___ has an old edit so flush to free WALs after random delay 130255 ms 2022-03-05 06:37:15,796 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5. because 7534b3b1812808d2d3c16342fbf347a5/e has an old edit so flush to free WALs after random delay 116322 ms 2022-03-05 06:37:26,230 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb. because 9b121c22e18472616debd0996ee2a4cb/e has an old edit so flush to free WALs after random delay 1204 ms 2022-03-05 06:37:27,434 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 9b121c22e18472616debd0996ee2a4cb 1/1 column families, dataSize=556.49 KB heapSize=1.00 MB 2022-03-05 06:37:27,491 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=556.49 KB at sequenceid=5821 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/.tmp/e/ce9d47626e08440181f7ae5ae1389f92 2022-03-05 06:37:27,508 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/ce9d47626e08440181f7ae5ae1389f92, entries=6444, sequenceid=5821, filesize=66.9 K 2022-03-05 06:37:27,509 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~556.49 KB/569846, heapSize ~1.00 MB/1053416, currentSize=0 B/0 for 9b121c22e18472616debd0996ee2a4cb in 75ms, sequenceid=5821, compaction requested=false 2022-03-05 06:38:00,254 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 644 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:38:00,254 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 590 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:38:13,719 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 877 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:38:17,178 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb. because ccdacf4b414cb8d1bc548efc77c663cb/e has an old edit so flush to free WALs after random delay 199077 ms 2022-03-05 06:38:17,677 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2718 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:38:17,677 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2671 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:38:17,677 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2043 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:38:23,460 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1213 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:38:26,689 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 501 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:38:36,193 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 619 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:39:12,119 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 7534b3b1812808d2d3c16342fbf347a5 1/1 column families, dataSize=171.61 KB heapSize=317.75 KB 2022-03-05 06:39:15,574 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:39:15,575 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:39:15,575 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:39:15,575 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:39:15,575 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:39:15,576 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:39:15,576 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:39:15,576 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:39:15,576 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:39:16,053 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 4333b489291b1f77671b8f2e78779a1e 1/1 column families, dataSize=7.24 KB heapSize=11.58 KB 2022-03-05 06:39:21,610 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4613ms No GCs detected 2022-03-05 06:39:23,295 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1184ms No GCs detected 2022-03-05 06:39:25,266 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646461987115 with entries=833, filesize=361.58 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646462355575 2022-03-05 06:39:25,267 WARN [Close-WAL-Writer-26] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:39:25,267 WARN [Close-WAL-Writer-26] wal.AsyncProtobufLogWriter: normal close failed, try recover java.lang.IllegalStateException: should call flush first before calling close at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:510) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:514) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:39:25,267 INFO [Close-WAL-Writer-26] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646461987115 2022-03-05 06:39:25,268 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646462347892","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":17376,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":97,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:39:25,268 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646462349368","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":15900,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":6,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:39:25,268 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646462351265","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":14003,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:39:25,269 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646462354955","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":10314,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:39:25,269 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646462352325","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":12944,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:39:25,269 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646462350525","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":14744,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:39:25,269 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646462354914","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":10355,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:39:25,268 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646462351147","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":14121,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:39:25,269 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646462353328","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":11941,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":1003,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:39:25,281 INFO [Close-WAL-Writer-26] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646461987115 after 14ms 2022-03-05 06:39:25,356 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646461759654.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646462365267.meta 2022-03-05 06:39:25,358 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646461759654.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646461759654.meta 2022-03-05 06:39:26,894 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=171.61 KB at sequenceid=3510 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/.tmp/e/33d987fbca6d43c9adf3b87135d47360 2022-03-05 06:39:27,965 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=7.24 KB at sequenceid=259 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/.tmp/__status_family___/060dac9534704ce8a83b134c970c0122 2022-03-05 06:39:36,113 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3403ms No GCs detected 2022-03-05 06:39:40,380 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.sync(AsyncProtobufLogWriter.java:148) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:394) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.consume(AsyncFSWAL.java:581) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:39:44,217 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:39:44,218 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:39:44,218 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:39:44,218 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:39:44,218 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:39:44,218 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:39:44,218 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:39:46,007 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1281ms No GCs detected 2022-03-05 06:39:46,214 INFO [Close-WAL-Writer-26] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646461987115 after 20947ms 2022-03-05 06:39:46,265 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/33d987fbca6d43c9adf3b87135d47360, entries=1965, sequenceid=3510, filesize=23.9 K 2022-03-05 06:39:46,268 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/060dac9534704ce8a83b134c970c0122, entries=3, sequenceid=259, filesize=5.1 K 2022-03-05 06:39:46,427 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646462365267.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646462376113.meta 2022-03-05 06:39:46,427 WARN [Close-WAL-Writer-30] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:39:46,428 WARN [Close-WAL-Writer-30] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:39:46,428 INFO [Close-WAL-Writer-30] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646462365267.meta 2022-03-05 06:39:46,432 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646462365267.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646462365267.meta 2022-03-05 06:39:46,462 INFO [Close-WAL-Writer-30] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646462365267.meta after 34ms 2022-03-05 06:39:46,719 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646462355575 with entries=58, filesize=24.53 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646462386427 2022-03-05 06:39:46,719 WARN [Close-WAL-Writer-27] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:39:46,719 WARN [Close-WAL-Writer-27] wal.AsyncProtobufLogWriter: normal close failed, try recover java.lang.IllegalStateException: should call flush first before calling close at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:510) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:514) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:39:46,720 INFO [Close-WAL-Writer-27] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646462355575 2022-03-05 06:39:46,723 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646462372113","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":14610,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:39:46,723 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646462376133","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":10590,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":8,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:39:46,723 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646462376145","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":10578,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":4032,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:39:46,724 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~171.61 KB/175730, heapSize ~317.73 KB/325360, currentSize=1.26 KB/1289 for 7534b3b1812808d2d3c16342fbf347a5 in 34605ms, sequenceid=3510, compaction requested=true 2022-03-05 06:39:46,725 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~7.24 KB/7417, heapSize ~11.56 KB/11840, currentSize=0 B/0 for 4333b489291b1f77671b8f2e78779a1e in 30671ms, sequenceid=259, compaction requested=true 2022-03-05 06:39:46,725 INFO [regionserver/cvp328:16201-longCompactions-0] regionserver.HRegion: Starting compaction of 7534b3b1812808d2d3c16342fbf347a5/e in aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5. 2022-03-05 06:39:46,725 INFO [regionserver/cvp328:16201-longCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/df10043559a04b41a80b794f69d00508, hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/772c759958594202a2f49aabc8e07210, hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/33d987fbca6d43c9adf3b87135d47360] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/.tmp, totalSize=115.7 K 2022-03-05 06:39:46,726 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 4333b489291b1f77671b8f2e78779a1e/__status_family___ in __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e. 2022-03-05 06:39:46,726 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/5ae68257a6de4ac9a050e260e638642c, hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/cc43a21e737c4039859495e2f7875c54, hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/060dac9534704ce8a83b134c970c0122] into tmpdir=hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/.tmp, totalSize=15.3 K 2022-03-05 06:39:46,940 INFO [Close-WAL-Writer-27] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646462355575 after 220ms 2022-03-05 06:39:48,229 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 4333b489291b1f77671b8f2e78779a1e#__status_family___#compaction#28 average throughput is 0.36 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 06:39:48,472 INFO [regionserver/cvp328:16201-longCompactions-0] throttle.PressureAwareThroughputController: 7534b3b1812808d2d3c16342fbf347a5#e#compaction#29 average throughput is 30.53 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 06:39:49,037 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 4333b489291b1f77671b8f2e78779a1e/__status_family___ of 4333b489291b1f77671b8f2e78779a1e into bb47857b71e44063ae29d154cb9b32a9(size=5.2 K), total size for store is 5.2 K. This selection was in queue for 0sec, and took 2sec to execute. 2022-03-05 06:39:49,037 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=__test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., storeName=4333b489291b1f77671b8f2e78779a1e/__status_family___, priority=13, startTime=1646462386725; duration=2sec 2022-03-05 06:39:49,043 INFO [regionserver/cvp328:16201-longCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 7534b3b1812808d2d3c16342fbf347a5/e of 7534b3b1812808d2d3c16342fbf347a5 into 26b6ff8862b947f38dd508b795e21c3d(size=112.6 K), total size for store is 112.6 K. This selection was in queue for 0sec, and took 2sec to execute. 2022-03-05 06:39:49,043 INFO [regionserver/cvp328:16201-longCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., storeName=7534b3b1812808d2d3c16342fbf347a5/e, priority=13, startTime=1646462386724; duration=2sec 2022-03-05 06:39:50,465 WARN [Close-WAL-Writer-30] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646462365267.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646462365267.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 06:39:50,950 INFO [Close-WAL-Writer-27] util.RecoverLeaseFSUtils: Recovered lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646462355575 after 4230ms 2022-03-05 06:40:15,572 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 7d1b53259f1d8280a47f7704f98f2c9f 1/1 column families, dataSize=685.74 KB heapSize=1.15 MB 2022-03-05 06:40:18,550 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 534 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:40:21,399 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1072ms No GCs detected 2022-03-05 06:40:22,856 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=685.74 KB at sequenceid=4474 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/.tmp/e/5dfc562018a54a118fecf49a66576877 2022-03-05 06:40:22,912 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/5dfc562018a54a118fecf49a66576877, entries=6612, sequenceid=4474, filesize=65.7 K 2022-03-05 06:40:22,914 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~685.74 KB/702195, heapSize ~1.15 MB/1204456, currentSize=0 B/0 for 7d1b53259f1d8280a47f7704f98f2c9f in 7342ms, sequenceid=4474, compaction requested=false 2022-03-05 06:41:36,255 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing ccdacf4b414cb8d1bc548efc77c663cb 1/1 column families, dataSize=297.95 KB heapSize=514.72 KB 2022-03-05 06:41:37,704 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=297.95 KB at sequenceid=2465 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/.tmp/e/fcdb7dada2f04aeba960d89d30050603 2022-03-05 06:41:37,860 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/fcdb7dada2f04aeba960d89d30050603, entries=2942, sequenceid=2465, filesize=30.8 K 2022-03-05 06:41:37,879 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~297.95 KB/305099, heapSize ~514.70 KB/527056, currentSize=181 B/181 for ccdacf4b414cb8d1bc548efc77c663cb in 1624ms, sequenceid=2465, compaction requested=false 2022-03-05 06:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=301.29 KB, freeSize=306.91 MB, max=307.20 MB, blockCount=9, accesses=4959, hits=4803, hitRatio=96.85%, , cachingAccesses=4819, cachingHits=4798, cachingHitsRatio=99.56%, evictions=1118, evicted=0, evictedPerRun=0.0 2022-03-05 06:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 06:42:11,161 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 636 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:42:12,521 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1200 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:42:26,014 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391. because 0a764e51258807790269e46c55b99391/e has an old edit so flush to free WALs after random delay 214195 ms 2022-03-05 06:42:29,993 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2272ms No GCs detected 2022-03-05 06:42:30,001 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3831 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:42:30,001 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2768 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:42:34,549 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 7122 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:42:34,571 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6657 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:42:34,571 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6636 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:42:34,571 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6436 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:42:34,571 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5898 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:42:34,571 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4895 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:42:34,571 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4852 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:42:34,572 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4169 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:42:34,572 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3765 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:42:34,572 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3421 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:42:34,572 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3261 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:42:34,572 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2386 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:42:34,572 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1754 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:42:44,468 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1503ms No GCs detected 2022-03-05 06:42:48,652 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 8084 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:42:48,849 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 7425 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:42:48,849 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 7082 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:42:48,849 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6878 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:42:48,849 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5924 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:42:56,887 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 890 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:42:56,887 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 724 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:43:00,006 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2559 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:43:00,006 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2090 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:43:00,006 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2081 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:43:00,006 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1505 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:43:08,652 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: rg.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 06:43:09,791 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646462376113.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646462588649.meta 2022-03-05 06:43:09,791 WARN [Close-WAL-Writer-31] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:43:09,791 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646462376113.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646462376113.meta 2022-03-05 06:43:09,791 WARN [Close-WAL-Writer-31] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:43:09,791 INFO [Close-WAL-Writer-31] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646462376113.meta 2022-03-05 06:43:09,805 INFO [Close-WAL-Writer-31] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646462376113.meta after 14ms 2022-03-05 06:43:11,907 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 544 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:43:12,740 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 787 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:43:13,865 WARN [Close-WAL-Writer-31] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646462376113.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646462376113.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 06:43:14,863 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1765 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:43:25,224 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:43:25,225 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:43:25,225 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:43:25,225 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:43:25,225 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:43:25,225 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:43:31,750 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646462386427 with entries=508, filesize=223.55 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646462605225 2022-03-05 06:43:31,750 WARN [Close-WAL-Writer-28] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:43:31,750 WARN [Close-WAL-Writer-28] wal.AsyncProtobufLogWriter: normal close failed, try recover java.lang.IllegalStateException: should call flush first before calling close at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:510) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:514) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:43:31,751 INFO [Close-WAL-Writer-28] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646462386427 2022-03-05 06:43:31,751 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646462595966","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":15785,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":4,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:43:31,751 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646462599264","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":12487,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:43:31,751 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646462596750","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":15001,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":788,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:43:31,751 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646462597987","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":13764,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":1237,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:43:31,751 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646462600187","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":11564,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:43:32,792 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1040 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:43:32,792 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1040 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:43:32,792 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 579 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:43:32,796 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646462597988","responsesize":"104","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 15 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x06Kernel ","processingtimems":14808,"client":"172.31.0.219:43118","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":15} 2022-03-05 06:43:33,949 INFO [Close-WAL-Writer-28] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646462386427 after 2198ms 2022-03-05 06:43:34,444 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 668 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:43:36,513 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1755 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:43:38,763 INFO [Close-WAL-Writer-28] retry.RetryInvocationHandler: .apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.recoverLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 06:43:38,767 INFO [Close-WAL-Writer-28] util.RecoverLeaseFSUtils: Recovered lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646462386427 after 7016ms 2022-03-05 06:43:44,077 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646462588649.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646462621750.meta 2022-03-05 06:43:44,077 WARN [Close-WAL-Writer-31] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:43:44,077 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646462588649.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646462588649.meta 2022-03-05 06:43:44,078 WARN [Close-WAL-Writer-31] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:43:44,078 INFO [Close-WAL-Writer-31] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646462588649.meta 2022-03-05 06:43:44,136 INFO [Close-WAL-Writer-31] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646462588649.meta after 58ms 2022-03-05 06:43:48,138 WARN [Close-WAL-Writer-31] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646462588649.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646462588649.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 06:44:27,322 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 939 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:44:27,404 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 835 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:44:27,404 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 714 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:44:38,936 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 740 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:44:53,660 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 564 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:44:53,777 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 681 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:45:04,936 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4765 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:45:08,966 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 7828 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:45:08,966 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 7784 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:45:08,966 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 7243 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:45:08,966 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 7243 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:45:08,966 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5948 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:45:08,909 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1449ms No GCs detected 2022-03-05 06:45:13,990 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4945 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:45:13,990 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4918 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:45:13,990 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3625 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:45:13,990 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3172 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:45:13,990 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2669 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:45:15,586 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4265 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:45:15,586 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3969 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:45:15,586 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3590 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:45:15,586 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3582 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:45:15,586 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2717 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:45:15,586 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 941 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:45:15,586 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 696 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:45:15,587 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 621 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:45:15,641 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646462700899","responsesize":"84","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 12 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC ","processingtimems":14742,"client":"172.30.41.118:54986","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":12} 2022-03-05 06:45:17,953 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 512 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:45:25,731 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 783 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:45:28,121 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 870 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:45:39,718 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 512 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:45:46,278 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 577 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:46:00,210 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 0a764e51258807790269e46c55b99391 1/1 column families, dataSize=945.50 KB heapSize=1.66 MB 2022-03-05 06:46:02,136 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1726 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:46:02,136 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1460 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:46:04,186 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3204 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:46:04,186 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3196 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:46:04,186 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2711 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:46:04,186 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2049 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:46:04,186 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2032 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:46:04,186 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1880 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:46:04,186 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1420 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:46:04,186 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1298 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:46:04,546 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=945.50 KB at sequenceid=7546 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/.tmp/e/0333a2bb2b2147d192b32ccff4a351ca 2022-03-05 06:46:04,572 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/0333a2bb2b2147d192b32ccff4a351ca, entries=10164, sequenceid=7546, filesize=92.2 K 2022-03-05 06:46:04,573 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~945.50 KB/968197, heapSize ~1.66 MB/1739808, currentSize=1.49 KB/1529 for 0a764e51258807790269e46c55b99391 in 4363ms, sequenceid=7546, compaction requested=false 2022-03-05 06:46:15,307 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2239ms No GCs detected 2022-03-05 06:46:19,365 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:46:19,487 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:46:19,487 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:46:19,487 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:46:19,487 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:46:19,487 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:46:19,487 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:46:20,322 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646462605225 with entries=401, filesize=181.74 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646462779491 2022-03-05 06:46:20,322 WARN [Close-WAL-Writer-29] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:46:20,322 WARN [Close-WAL-Writer-29] wal.AsyncProtobufLogWriter: normal close failed, try recover java.lang.IllegalStateException: should call flush first before calling close at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:510) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:514) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:46:20,323 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646458473563 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646458473563 2022-03-05 06:46:20,323 INFO [Close-WAL-Writer-29] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646462605225 2022-03-05 06:46:20,602 INFO [Close-WAL-Writer-29] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646462605225 after 279ms 2022-03-05 06:46:20,611 INFO [Log-Archiver-0] retry.RetryInvocationHandler: 040) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.rename(ClientNamenodeProtocolServerSideTranslatorPB.java:652) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.rename over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 06:46:20,905 INFO [Log-Archiver-0] retry.RetryInvocationHandler: lPB.ClientNamenodeProtocolServerSideTranslatorPB.rename(ClientNamenodeProtocolServerSideTranslatorPB.java:652) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.rename over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 1 failover attempts. Trying to failover after sleeping for 661ms. 2022-03-05 06:46:21,567 INFO [Log-Archiver-0] retry.RetryInvocationHandler: PB.ClientNamenodeProtocolServerSideTranslatorPB.rename(ClientNamenodeProtocolServerSideTranslatorPB.java:652) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.rename over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 2 failover attempts. Trying to failover after sleeping for 1569ms. 2022-03-05 06:46:25,098 INFO [Close-WAL-Writer-29] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646462605225 after 4775ms 2022-03-05 06:46:25,301 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646459081967 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646459081967 2022-03-05 06:46:26,246 INFO [Close-WAL-Writer-29] retry.RetryInvocationHandler: apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.isFileClosed(ClientNamenodeProtocolServerSideTranslatorPB.java:1322) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.isFileClosed over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 06:46:26,247 INFO [Close-WAL-Writer-29] retry.RetryInvocationHandler: enodeProtocolServerSideTranslatorPB.isFileClosed(ClientNamenodeProtocolServerSideTranslatorPB.java:1322) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.isFileClosed over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 340ms. 2022-03-05 06:46:26,418 INFO [Log-Archiver-0] retry.RetryInvocationHandler: at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setTimes(ClientNamenodeProtocolServerSideTranslatorPB.java:1062) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.setTimes over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 06:46:26,418 WARN [Log-Archiver-0] retry.RetryInvocationHandler: A failover has occurred since the start of call #2299 ClientNamenodeProtocolTranslatorPB.setTimes over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 2022-03-05 06:46:26,419 INFO [Log-Archiver-0] retry.RetryInvocationHandler: lientNamenodeProtocolServerSideTranslatorPB.setTimes(ClientNamenodeProtocolServerSideTranslatorPB.java:1062) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.setTimes over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 887ms. 2022-03-05 06:46:27,232 INFO [Close-WAL-Writer-29] retry.RetryInvocationHandler: enodeProtocolServerSideTranslatorPB.isFileClosed(ClientNamenodeProtocolServerSideTranslatorPB.java:1322) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.isFileClosed over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 2 failover attempts. Trying to failover after sleeping for 688ms. 2022-03-05 06:46:27,306 WARN [Log-Archiver-0] retry.RetryInvocationHandler: A failover has occurred since the start of call #2299 ClientNamenodeProtocolTranslatorPB.setTimes over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 2022-03-05 06:46:27,970 INFO [Log-Archiver-0] retry.RetryInvocationHandler: ientNamenodeProtocolServerSideTranslatorPB.setTimes(ClientNamenodeProtocolServerSideTranslatorPB.java:1062) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.setTimes over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 2 failover attempts. Trying to failover after sleeping for 1420ms. 2022-03-05 06:46:29,390 WARN [Log-Archiver-0] retry.RetryInvocationHandler: A failover has occurred since the start of call #2299 ClientNamenodeProtocolTranslatorPB.setTimes over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 2022-03-05 06:46:31,742 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646459115250 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646459115250 2022-03-05 06:46:32,233 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4106 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:46:32,234 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3610 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:46:32,234 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2632 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:46:32,234 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2524 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:46:32,234 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1830 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:46:32,234 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1466 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:46:32,234 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1104 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:46:32,234 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 586 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:46:32,439 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646459236812 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646459236812 2022-03-05 06:46:33,338 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 525 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:46:34,315 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646459278076 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646459278076 2022-03-05 06:46:36,530 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1210ms No GCs detected 2022-03-05 06:46:36,601 INFO [Close-WAL-Writer-29] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=2 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646462605225 after 16278ms 2022-03-05 06:46:36,627 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646459636103 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646459636103 2022-03-05 06:46:36,663 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646460236510 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646460236510 2022-03-05 06:46:36,676 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646460785715 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646460785715 2022-03-05 06:46:36,684 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646461389438 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646461389438 2022-03-05 06:46:36,704 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646461481883 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646461481883 2022-03-05 06:46:36,721 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646461542624 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646461542624 2022-03-05 06:46:36,734 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646461743554 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646461743554 2022-03-05 06:46:36,746 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646461766910 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646461766910 2022-03-05 06:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=301.29 KB, freeSize=306.91 MB, max=307.20 MB, blockCount=9, accesses=4981, hits=4825, hitRatio=96.87%, , cachingAccesses=4841, cachingHits=4820, cachingHitsRatio=99.57%, evictions=1147, evicted=0, evictedPerRun=0.0 2022-03-05 06:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 06:46:48,339 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 909 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:46:59,506 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2114 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:47:00,412 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2562 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:47:00,438 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2500 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:47:00,438 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2022 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:47:00,438 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1429 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:47:00,451 INFO [Close-WAL-Writer-29] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=3 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646462605225 after 40128ms 2022-03-05 06:47:00,451 WARN [Close-WAL-Writer-29] util.RecoverLeaseFSUtils: Cannot recoverLease after trying for 23000ms (hbase.lease.recovery.timeout); continuing, but may be DATALOSS!!!; attempt=3 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646462605225 after 40128ms 2022-03-05 06:47:09,284 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 538 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:47:31,322 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 556 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:47:31,322 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 554 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:47:57,007 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1537ms No GCs detected 2022-03-05 06:47:57,057 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4844 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:47:57,057 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3998 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:47:57,057 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3869 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:47:57,057 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2139 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:48:11,414 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 7261 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:48:11,414 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6887 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:48:11,414 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5429 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:48:11,414 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3597 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:48:11,414 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3210 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:48:11,414 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1995 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:48:16,987 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 849 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:48:16,987 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 749 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:48:25,742 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2658 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:48:25,742 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2500 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:48:25,742 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 828 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:48:25,743 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 500 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:48:28,283 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 509 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:48:28,283 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 507 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:48:46,834 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 557 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:48:54,684 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1234 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:48:54,684 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 958 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:49:05,035 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2023ms No GCs detected 2022-03-05 06:49:07,428 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1892ms No GCs detected 2022-03-05 06:49:08,247 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:49:10,777 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:49:10,778 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:49:10,778 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:49:10,778 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:49:10,778 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:49:11,011 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646462779491 with entries=361, filesize=161.33 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646462950778 2022-03-05 06:49:11,011 WARN [Close-WAL-Writer-30] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:49:11,013 WARN [Close-WAL-Writer-30] wal.AsyncProtobufLogWriter: normal close failed, try recover java.lang.IllegalStateException: should call flush first before calling close at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:510) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:514) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:49:11,014 INFO [Close-WAL-Writer-30] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646462779491 2022-03-05 06:49:11,015 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646462940276","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":10738,"client":"172.30.41.155:52590","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:49:11,247 INFO [Close-WAL-Writer-30] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646462779491 after 232ms 2022-03-05 06:49:15,677 INFO [Close-WAL-Writer-30] retry.RetryInvocationHandler: .apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.recoverLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 06:49:16,537 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 882 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:49:16,537 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 866 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:49:19,967 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2003 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:49:24,170 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6206 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 06:49:20,290 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1037ms No GCs detected 2022-03-05 06:49:31,993 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Connection to 172.30.41.155/172.30.41.155:15010 closed at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$channelInactive$2(FanOutOneBlockAsyncDFSOutput.java:286) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.channelInactive(FanOutOneBlockAsyncDFSOutput.java:285) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:389) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:354) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelInactive(IdleStateHandler.java:277) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:831) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:49:35,257 INFO [Close-WAL-Writer-30] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646462779491 after 24242ms 2022-03-05 06:49:35,257 WARN [Close-WAL-Writer-30] util.RecoverLeaseFSUtils: Cannot recoverLease after trying for 23000ms (hbase.lease.recovery.timeout); continuing, but may be DATALOSS!!!; attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646462779491 after 24242ms 2022-03-05 06:49:35,815 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646462950778 with entries=56, filesize=22.91 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646462974968 2022-03-05 06:49:35,815 WARN [Close-WAL-Writer-30] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:49:35,815 WARN [Close-WAL-Writer-30] wal.AsyncProtobufLogWriter: normal close failed, try recover java.lang.IllegalStateException: should call flush first before calling close at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:510) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:514) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:49:35,816 INFO [Close-WAL-Writer-30] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646462950778 2022-03-05 06:49:35,818 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646462963116","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":12702,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:49:36,190 INFO [Close-WAL-Writer-30] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646462950778 after 374ms 2022-03-05 06:49:40,191 INFO [Close-WAL-Writer-30] util.RecoverLeaseFSUtils: Recovered lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646462950778 after 4375ms 2022-03-05 06:49:43,529 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2186 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:49:43,529 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1469 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:49:43,530 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1005ms No GCs detected 2022-03-05 06:49:48,594 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 687 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:49:48,594 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 652 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:49:59,093 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3124 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:49:59,093 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2332 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:49:59,093 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1372 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:49:59,093 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1223 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:49:59,093 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1024 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:50:00,578 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2811ms No GCs detected 2022-03-05 06:50:09,435 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5760 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:50:09,435 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 7811 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:50:09,435 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4062 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:50:09,438 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3968 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:50:09,438 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3968 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:50:09,438 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3954 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:50:09,442 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: rg.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 06:50:09,663 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: NamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 573ms. 2022-03-05 06:50:10,764 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: NamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 2 failover attempts. Trying to failover after sleeping for 690ms. 2022-03-05 06:50:11,456 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: amenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 3 failover attempts. Trying to failover after sleeping for 3256ms. 2022-03-05 06:50:12,817 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 06:50:12,819 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 687 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:50:12,819 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 685 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:50:14,712 WARN [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: A failover has occurred since the start of call #2369 ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 2022-03-05 06:50:14,715 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: amenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 4 failover attempts. Trying to failover after sleeping for 3657ms. 2022-03-05 06:50:15,647 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 716 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:50:15,647 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 575 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:50:18,568 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646462621750.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646462996118.meta 2022-03-05 06:50:18,568 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646462621750.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646462621750.meta 2022-03-05 06:50:18,568 WARN [Close-WAL-Writer-32] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:50:18,568 WARN [Close-WAL-Writer-32] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:50:18,569 INFO [Close-WAL-Writer-32] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646462621750.meta 2022-03-05 06:50:18,633 INFO [Log-Archiver-0] retry.RetryInvocationHandler: at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setTimes(ClientNamenodeProtocolServerSideTranslatorPB.java:1062) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.setTimes over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 06:50:18,635 INFO [Close-WAL-Writer-32] retry.RetryInvocationHandler: .apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.recoverLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 06:50:18,635 WARN [Close-WAL-Writer-32] retry.RetryInvocationHandler: A failover has occurred since the start of call #2374 ClientNamenodeProtocolTranslatorPB.recoverLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 2022-03-05 06:50:18,635 INFO [Log-Archiver-0] retry.RetryInvocationHandler: lientNamenodeProtocolServerSideTranslatorPB.setTimes(ClientNamenodeProtocolServerSideTranslatorPB.java:1062) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.setTimes over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 1 failover attempts. Trying to failover after sleeping for 498ms. 2022-03-05 06:50:18,640 INFO [Close-WAL-Writer-32] retry.RetryInvocationHandler: menodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.recoverLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 1 failover attempts. Trying to failover after sleeping for 301ms. 2022-03-05 06:50:18,960 INFO [Close-WAL-Writer-32] retry.RetryInvocationHandler: menodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.recoverLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 2 failover attempts. Trying to failover after sleeping for 701ms. 2022-03-05 06:50:19,136 WARN [Log-Archiver-0] retry.RetryInvocationHandler: A failover has occurred since the start of call #2373 ClientNamenodeProtocolTranslatorPB.setTimes over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 2022-03-05 06:50:19,742 INFO [Log-Archiver-0] retry.RetryInvocationHandler: lientNamenodeProtocolServerSideTranslatorPB.setTimes(ClientNamenodeProtocolServerSideTranslatorPB.java:1062) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.setTimes over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 2 failover attempts. Trying to failover after sleeping for 879ms. 2022-03-05 06:50:20,621 WARN [Log-Archiver-0] retry.RetryInvocationHandler: A failover has occurred since the start of call #2373 ClientNamenodeProtocolTranslatorPB.setTimes over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 2022-03-05 06:50:20,752 INFO [Close-WAL-Writer-32] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646462621750.meta after 2183ms 2022-03-05 06:50:24,753 WARN [Close-WAL-Writer-32] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646462621750.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646462621750.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 06:50:35,873 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1066 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:50:42,650 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 647 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:50:48,248 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 815 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:51:06,609 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1037 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:51:18,005 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 568 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:51:32,924 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 503 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:51:32,925 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 503 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=301.29 KB, freeSize=306.91 MB, max=307.20 MB, blockCount=9, accesses=5003, hits=4847, hitRatio=96.88%, , cachingAccesses=4863, cachingHits=4842, cachingHitsRatio=99.57%, evictions=1177, evicted=0, evictedPerRun=0.0 2022-03-05 06:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 06:52:30,337 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 737 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:52:30,337 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 673 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:54:14,364 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1246 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:54:36,181 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 576 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:54:47,791 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1487 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:54:48,810 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1311 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:54:48,810 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 874 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:55:00,538 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 938 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:55:00,538 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 720 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:55:00,538 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 720 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:55:11,901 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 639 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:55:22,944 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 719 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:55:56,259 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1395 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:56:30,960 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 570 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:56:31,573 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 578 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:56:34,718 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1250 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:56:34,719 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 901 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:56:34,719 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 899 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:56:34,719 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 899 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:56:34,719 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 898 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:56:34,791 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 970 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:56:34,791 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 634 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:56:44,503 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=301.29 KB, freeSize=306.91 MB, max=307.20 MB, blockCount=9, accesses=5022, hits=4866, hitRatio=96.89%, , cachingAccesses=4882, cachingHits=4861, cachingHitsRatio=99.57%, evictions=1207, evicted=0, evictedPerRun=0.0 2022-03-05 06:56:44,515 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:56:44,515 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1524ms No GCs detected 2022-03-05 06:56:44,517 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:56:44,517 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:56:44,517 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:56:44,517 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:56:44,517 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.sync(AsyncProtobufLogWriter.java:148) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:394) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.consume(AsyncFSWAL.java:581) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:56:46,975 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646462974968 with entries=953, filesize=416.29 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646463404518 2022-03-05 06:56:46,975 WARN [Close-WAL-Writer-31] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:56:46,976 WARN [Close-WAL-Writer-31] wal.AsyncProtobufLogWriter: normal close failed, try recover java.lang.IllegalStateException: should call flush first before calling close at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:510) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:514) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:56:46,976 INFO [Close-WAL-Writer-31] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646462974968 2022-03-05 06:56:46,979 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646463396161","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":10817,"client":"172.30.41.155:52590","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:56:46,979 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646463396304","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":10674,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":662,"class":"HRegionServer","multi.mutations":1} 2022-03-05 06:56:48,218 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 06:56:48,269 INFO [Close-WAL-Writer-31] retry.RetryInvocationHandler: .apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.recoverLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 06:56:48,273 INFO [Close-WAL-Writer-31] retry.RetryInvocationHandler: menodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.recoverLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 400ms. 2022-03-05 06:56:48,678 INFO [Close-WAL-Writer-31] retry.RetryInvocationHandler: enodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.recoverLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 2 failover attempts. Trying to failover after sleeping for 1581ms. 2022-03-05 06:56:51,546 INFO [Close-WAL-Writer-31] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646462974968 after 4570ms 2022-03-05 06:56:53,548 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1263 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:56:55,557 INFO [Close-WAL-Writer-31] util.RecoverLeaseFSUtils: Recovered lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646462974968 after 8581ms 2022-03-05 06:57:05,881 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1674 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:57:12,015 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1221ms No GCs detected 2022-03-05 06:57:17,644 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6887 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:57:17,644 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5663 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:57:17,644 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5662 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:57:17,644 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3622 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:57:17,644 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2130 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:57:17,644 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1128 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:57:21,097 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3097 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:57:23,156 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 876 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:57:28,728 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2465 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:57:28,729 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2461 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:57:28,729 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1346 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:57:32,235 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1100ms No GCs detected 2022-03-05 06:57:35,373 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6649 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:57:35,386 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6655 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:57:35,386 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6655 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:57:35,386 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6563 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:57:35,386 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5231 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:57:35,386 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5231 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:57:35,386 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3149 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:57:36,836 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1449 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:57:36,836 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1449 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:57:37,630 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2202 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:57:37,630 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1073 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:57:38,107 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 544 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:57:40,946 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2838 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:57:41,113 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: rg.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 06:57:41,130 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: NamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 1 failover attempts. Trying to failover after sleeping for 709ms. 2022-03-05 06:57:42,099 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: amenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 2 failover attempts. Trying to failover after sleeping for 1560ms. 2022-03-05 06:57:43,029 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 890 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:57:43,975 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646462996118.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646463457403.meta 2022-03-05 06:57:43,975 WARN [Close-WAL-Writer-33] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:57:43,975 WARN [Close-WAL-Writer-33] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 06:57:43,975 INFO [Close-WAL-Writer-33] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646462996118.meta 2022-03-05 06:57:43,976 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646462996118.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646462996118.meta 2022-03-05 06:57:44,036 INFO [Close-WAL-Writer-33] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646462996118.meta after 61ms 2022-03-05 06:57:44,132 INFO [Log-Archiver-0] retry.RetryInvocationHandler: 040) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.rename(ClientNamenodeProtocolServerSideTranslatorPB.java:652) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.rename over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 06:57:44,135 INFO [Log-Archiver-0] retry.RetryInvocationHandler: lPB.ClientNamenodeProtocolServerSideTranslatorPB.rename(ClientNamenodeProtocolServerSideTranslatorPB.java:652) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.rename over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 835ms. 2022-03-05 06:57:44,993 INFO [Log-Archiver-0] retry.RetryInvocationHandler: lPB.ClientNamenodeProtocolServerSideTranslatorPB.rename(ClientNamenodeProtocolServerSideTranslatorPB.java:652) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.rename over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 2 failover attempts. Trying to failover after sleeping for 853ms. 2022-03-05 06:57:47,958 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 514 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 06:57:48,038 WARN [Close-WAL-Writer-33] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646462996118.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646462996118.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 07:00:34,223 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2662 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:00:35,075 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2090 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:00:35,119 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 930 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:00:45,536 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2482ms No GCs detected 2022-03-05 07:00:47,778 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6685 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:00:47,780 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3247 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:00:49,782 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 07:01:00,887 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 586 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:01:16,288 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 536 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:01:20,292 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 07:01:28,914 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1381 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:01:28,914 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1067 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:01:28,914 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 991 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:01:28,914 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 794 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:01:28,914 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 587 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=301.29 KB, freeSize=306.91 MB, max=307.20 MB, blockCount=9, accesses=5047, hits=4891, hitRatio=96.91%, , cachingAccesses=4907, cachingHits=4886, cachingHitsRatio=99.57%, evictions=1237, evicted=0, evictedPerRun=0.0 2022-03-05 07:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 07:01:53,325 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1074 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:02:04,296 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 514 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:02:13,150 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2470 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:02:14,172 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2865 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:02:14,174 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2122 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:02:26,881 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1978 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:02:26,881 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1953 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:02:26,881 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 836 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:02:26,881 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 668 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:02:38,868 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 682 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:02:41,857 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1230 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:02:50,718 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2630ms No GCs detected 2022-03-05 07:02:53,827 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2204 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:02:53,836 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 579 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:02:53,836 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 575 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:02:53,836 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 575 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:03:00,455 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 566 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:03:02,513 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 848 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:03:02,513 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 816 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:04:11,119 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 557 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:04:43,870 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1094 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:04:43,870 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 715 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:05:01,957 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 666 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:05:07,024 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1581ms No GCs detected 2022-03-05 07:05:07,105 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4448 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:05:07,105 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4448 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:05:07,105 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4298 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:05:07,106 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4062 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:05:07,106 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1739 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:05:16,807 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1103 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:05:18,585 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1778 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:05:18,585 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1775 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:05:19,461 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1278ms No GCs detected 2022-03-05 07:05:20,490 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 542 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:05:21,265 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 775 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:05:30,423 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1003 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:05:30,470 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2340ms GC pool 'G1 Young Generation' had collection(s): count=1 time=1674ms 2022-03-05 07:05:30,484 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 881 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:05:30,484 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 823 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:05:42,560 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1327 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:05:44,117 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2382 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:05:44,117 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2122 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:05:53,427 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1196 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:05:56,008 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1072 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:05:56,025 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 971 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=301.29 KB, freeSize=306.91 MB, max=307.20 MB, blockCount=9, accesses=5069, hits=4913, hitRatio=96.92%, , cachingAccesses=4929, cachingHits=4908, cachingHitsRatio=99.57%, evictions=1267, evicted=0, evictedPerRun=0.0 2022-03-05 07:06:45,252 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646463404518 with entries=1320, filesize=576.96 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646464005228 2022-03-05 07:06:48,042 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 07:06:48,435 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1004 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:06:49,681 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 897 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:07:01,701 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 659 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:07:01,701 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 524 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:07:09,328 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 958 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:07:32,681 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 526 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:07:32,682 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 527 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:07:41,534 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1018 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:07:42,180 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 927 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:07:44,432 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2251 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:07:46,403 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646463457403.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646464065819.meta 2022-03-05 07:07:46,403 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646463457403.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646463457403.meta 2022-03-05 07:07:58,312 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3434 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:07:58,312 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3409 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:07:58,312 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2335 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:07:58,312 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1605 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:07:58,312 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1605 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:08:02,771 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 618 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:08:07,356 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1596ms No GCs detected 2022-03-05 07:08:08,748 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3980 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:08:08,748 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3149 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:08:08,748 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3146 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:08:08,749 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1028 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:08:11,059 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1868 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:08:16,411 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1030ms No GCs detected 2022-03-05 07:08:21,629 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5207 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:08:21,629 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5087 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:08:21,635 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2222ms No GCs detected 2022-03-05 07:08:23,552 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5634 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:08:23,553 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5560 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:08:23,553 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1922 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:08:31,675 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3584ms GC pool 'G1 Young Generation' had collection(s): count=1 time=3913ms 2022-03-05 07:08:32,830 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 07:08:33,490 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1331 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:08:33,490 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1331 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:08:33,491 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1330 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:08:33,491 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1328 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:08:33,497 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1335 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:08:33,497 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1335 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:08:33,497 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1335 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:08:33,497 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1335 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:08:33,497 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1335 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:08:33,497 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1334 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:08:33,497 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 662 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:08:33,497 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 662 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:08:37,836 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2245 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:08:37,836 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2177 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:08:39,756 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 580 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:08:41,325 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1069ms No GCs detected 2022-03-05 07:08:42,267 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1225 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:08:44,264 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1496ms No GCs detected 2022-03-05 07:08:45,551 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1286 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:08:45,551 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1053 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:08:45,551 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 633 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:08:45,551 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 541 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:08:46,896 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 914 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:08:46,896 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 686 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:08:47,335 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: rg.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 07:08:47,591 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646464065819.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646464127287.meta 2022-03-05 07:08:47,592 WARN [Close-WAL-Writer-34] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 07:08:47,592 WARN [Close-WAL-Writer-34] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 07:08:47,592 INFO [Close-WAL-Writer-34] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646464065819.meta 2022-03-05 07:08:47,595 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646464065819.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646464065819.meta 2022-03-05 07:08:47,706 INFO [Close-WAL-Writer-34] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646464065819.meta after 114ms 2022-03-05 07:08:51,708 WARN [Close-WAL-Writer-34] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646464065819.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646464065819.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 07:08:54,638 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2393 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:08:54,640 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1040 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:09:16,277 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 641 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:09:38,623 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 906 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:09:49,266 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3071 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:09:55,179 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 8982 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:09:56,900 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 9250 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:09:56,900 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 8663 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:09:56,900 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3605 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:09:56,901 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2168 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:10:00,323 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 715 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:10:01,521 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1913 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:10:01,521 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1877 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:10:01,521 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1341 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:10:03,750 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1547 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:10:10,279 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6528 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:10:10,291 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6526 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:10:10,291 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4694 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:10:10,354 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2547 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:10:10,354 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1436 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:10:10,354 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1153 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:10:10,496 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 07:10:12,628 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 843 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:10:12,628 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 639 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:10:17,648 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: rg.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 07:10:27,995 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 07:10:29,990 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 5788ms No GCs detected 2022-03-05 07:10:48,949 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1940ms No GCs detected 2022-03-05 07:11:02,524 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1076ms No GCs detected 2022-03-05 07:11:02,720 WARN [regionserver/cvp328:16201] util.Sleeper: We slept 15893ms instead of 5000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired 2022-03-05 07:11:12,855 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646464127287.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646464217644.meta 2022-03-05 07:11:12,855 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646464127287.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646464127287.meta 2022-03-05 07:11:12,856 WARN [Close-WAL-Writer-35] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 07:11:12,856 WARN [Close-WAL-Writer-35] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 07:11:12,856 INFO [Close-WAL-Writer-35] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646464127287.meta 2022-03-05 07:11:13,283 INFO [Close-WAL-Writer-35] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646464127287.meta after 427ms 2022-03-05 07:11:16,924 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646464005228 with entries=455, filesize=203.58 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646464272855 2022-03-05 07:11:16,925 WARN [Close-WAL-Writer-33] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 07:11:16,926 WARN [Close-WAL-Writer-33] wal.AsyncProtobufLogWriter: normal close failed, try recover java.lang.IllegalStateException: should call flush first before calling close at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:510) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:514) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 07:11:16,926 INFO [Close-WAL-Writer-33] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646464005228 2022-03-05 07:11:16,981 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646464223702","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":53279,"client":"172.30.41.155:52590","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 07:11:16,981 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646464229991","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":46990,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 07:11:16,981 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646464224952","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":52029,"client":"172.30.41.155:52590","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 07:11:16,981 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646464229997","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":46984,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":6,"class":"HRegionServer","multi.mutations":1} 2022-03-05 07:11:16,981 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646464230027","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":46954,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":3,"class":"HRegionServer","multi.mutations":1} 2022-03-05 07:11:16,982 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646464228610","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":48372,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 07:11:16,982 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646464228636","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":48346,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 07:11:16,983 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646464228622","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":48361,"client":"172.30.41.155:52590","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 07:11:17,015 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646464230237","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":46777,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 07:11:17,023 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646464229990","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":47033,"client":"172.30.41.155:52590","multi.service_calls":0,"queuetimems":1368,"class":"HRegionServer","multi.mutations":1} 2022-03-05 07:11:17,023 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646464229990","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":47033,"client":"172.30.41.155:52590","multi.service_calls":0,"queuetimems":1368,"class":"HRegionServer","multi.mutations":1} 2022-03-05 07:11:17,024 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646464229989","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":47035,"client":"172.30.41.155:52590","multi.service_calls":0,"queuetimems":1367,"class":"HRegionServer","multi.mutations":1} 2022-03-05 07:11:17,024 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646464229991","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":47033,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 07:11:17,024 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646464229997","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":47027,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":6,"class":"HRegionServer","multi.mutations":1} 2022-03-05 07:11:17,024 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646464229990","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":47034,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 07:11:17,059 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646464230237","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":46822,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 07:11:17,466 WARN [Close-WAL-Writer-35] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646464127287.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646464127287.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 07:11:17,899 INFO [Close-WAL-Writer-33] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646464005228 after 973ms 2022-03-05 07:11:18,959 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 501 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:11:20,209 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 734 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:11:20,209 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 734 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:11:20,209 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 734 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:11:21,286 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1075 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:11:21,286 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1066 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:11:23,052 INFO [Close-WAL-Writer-33] util.RecoverLeaseFSUtils: Recovered lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646464005228 after 6126ms 2022-03-05 07:11:24,213 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1161 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:11:44,752 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=301.29 KB, freeSize=306.91 MB, max=307.20 MB, blockCount=9, accesses=5116, hits=4960, hitRatio=96.95%, , cachingAccesses=4976, cachingHits=4955, cachingHitsRatio=99.58%, evictions=1296, evicted=0, evictedPerRun=0.0 2022-03-05 07:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 07:11:49,057 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1151 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:11:56,315 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3987 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:11:56,315 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3274 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:11:56,315 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2991 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:11:56,315 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1349 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:12:02,484 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1377 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:12:02,484 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1365 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:12:02,484 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1291 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:12:02,484 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1137 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:12:07,501 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1887 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:12:13,733 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1482ms No GCs detected 2022-03-05 07:12:16,475 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4583 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:12:16,475 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4582 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:12:17,796 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1318 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:12:20,245 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3547 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:12:20,245 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2803 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:12:20,245 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2449 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:12:20,245 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2282 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:12:20,245 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2251 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:12:27,035 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2068 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:12:27,036 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 984 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:12:27,036 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 723 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:12:27,036 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 619 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:12:44,876 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3166 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:12:44,876 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2883 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:12:44,876 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1998 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:13:30,535 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 924 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:13:30,535 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 907 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:13:36,640 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 802 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:14:39,235 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1495 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:14:39,236 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 688 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:15:34,948 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 675 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:15:53,695 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1429 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:15:53,695 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 606 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:15:58,700 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 769 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:15:58,700 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 563 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:16:09,795 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5614 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:16:09,795 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5356 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:16:09,871 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4159 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:16:10,636 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2413 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:16:10,636 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2306 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:16:10,642 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1374 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:16:18,327 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 916 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:16:24,589 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1493 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:16:25,251 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1404 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:16:36,582 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2556 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:16:38,621 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 731 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:16:38,621 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1408 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:16:38,621 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 860 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:16:39,999 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1370 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:16:41,971 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 07:16:41,979 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 1 failover attempts. Trying to failover after sleeping for 475ms. 2022-03-05 07:16:42,456 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: tNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 2 failover attempts. Trying to failover after sleeping for 1445ms. 2022-03-05 07:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=301.29 KB, freeSize=306.91 MB, max=307.20 MB, blockCount=9, accesses=5135, hits=4979, hitRatio=96.96%, , cachingAccesses=4995, cachingHits=4974, cachingHitsRatio=99.58%, evictions=1326, evicted=0, evictedPerRun=0.0 2022-03-05 07:16:44,541 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: tNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 3 failover attempts. Trying to failover after sleeping for 3512ms. 2022-03-05 07:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 07:16:56,395 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4188 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:16:56,395 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3359 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:16:56,395 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3133 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:16:56,395 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1507 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:16:56,395 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1479 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:16:57,278 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 874 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:16:57,278 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 665 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:17:00,357 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 779 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:17:00,358 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 779 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:17:00,358 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 779 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:17:00,358 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 684 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:17:02,250 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1776 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:17:02,250 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1621 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:17:02,250 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1453 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:17:02,251 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 979 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:17:02,252 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 876 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:17:02,252 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 697 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:17:09,801 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1125 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:17:11,538 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2767 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:17:18,436 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 527 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:17:30,895 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1203 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:17:30,895 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1203 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:17:41,240 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1583 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:17:48,963 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 07:19:23,503 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 897 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:20:20,018 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1398 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:20:22,826 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646464217644.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646464820144.meta 2022-03-05 07:20:22,826 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646464217644.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646464217644.meta 2022-03-05 07:20:22,994 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646464272855 with entries=1283, filesize=566.19 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646464822826 2022-03-05 07:20:30,544 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1917 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:20:30,544 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1000 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:20:30,544 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1000 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:20:30,883 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1271 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:20:31,565 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1020 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:20:31,565 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 992 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:20:31,565 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 763 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:20:33,069 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 879 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:20:33,069 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 879 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:20:35,291 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1586 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:20:36,890 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1389 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:20:38,149 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1242 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:20:52,944 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 07:20:57,448 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2511 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:20:57,449 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2486 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:21:31,441 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2812 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:21:31,442 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2812 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:21:31,442 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1905 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:21:31,442 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1805 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:21:31,442 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1043 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:21:31,442 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 641 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:21:31,442 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 602 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:21:42,704 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 786 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=301.29 KB, freeSize=306.91 MB, max=307.20 MB, blockCount=9, accesses=5160, hits=5004, hitRatio=96.98%, , cachingAccesses=5020, cachingHits=4999, cachingHitsRatio=99.58%, evictions=1356, evicted=0, evictedPerRun=0.0 2022-03-05 07:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 07:22:16,988 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 628 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:23:01,892 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1012ms No GCs detected 2022-03-05 07:23:02,687 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 793 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:23:02,687 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 791 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:23:02,687 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 788 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:23:47,091 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 738 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:24:00,166 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1341 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:24:00,166 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1340 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:24:00,315 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 748 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:24:00,315 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 626 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:24:00,315 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 625 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:25:18,364 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 563 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:25:23,449 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 977 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:25:35,083 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5529 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:25:35,083 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5524 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:25:35,083 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5391 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:25:35,084 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3929 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:25:35,230 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4041 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:25:35,231 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3952 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:25:35,231 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2769 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:25:35,231 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2769 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:25:35,231 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2693 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:25:35,231 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2471 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:25:35,231 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2465 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:25:35,231 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2217 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:25:35,231 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1593 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:25:47,545 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2526ms No GCs detected 2022-03-05 07:25:48,381 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5860 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:25:48,381 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5850 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:25:48,381 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4352 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:25:48,381 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1001 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:25:50,613 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 928 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:25:54,381 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2232 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:25:54,381 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2103 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:25:54,381 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1280 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:25:54,381 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1190 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:25:54,381 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1180 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:25:56,161 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 07:25:56,329 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 1 failover attempts. Trying to failover after sleeping for 780ms. 2022-03-05 07:25:57,110 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: tNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 2 failover attempts. Trying to failover after sleeping for 1763ms. 2022-03-05 07:26:00,041 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 517 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:26:28,523 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 611 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:26:39,687 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 513 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=301.29 KB, freeSize=306.91 MB, max=307.20 MB, blockCount=9, accesses=5182, hits=5026, hitRatio=96.99%, , cachingAccesses=5042, cachingHits=5021, cachingHitsRatio=99.58%, evictions=1385, evicted=0, evictedPerRun=0.0 2022-03-05 07:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 07:27:01,284 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3598 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:27:01,284 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3597 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:27:01,284 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3383 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:27:01,284 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3182 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:27:01,284 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2717 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:27:01,284 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2717 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:27:01,284 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1757 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:27:01,284 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1757 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:27:01,284 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1615 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:27:01,300 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 893 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:27:12,480 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 513 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:28:33,810 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 994 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:28:33,810 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 698 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:28:37,043 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3798 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:28:37,053 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3807 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:28:37,053 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3242 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:28:37,053 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3035 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:28:37,053 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3033 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:28:37,053 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3033 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:28:37,053 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2609 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:28:37,053 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1363 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:30:28,912 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1113 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:30:28,912 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 970 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:30:28,912 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 654 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:30:28,912 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 551 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:30:29,299 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646464820144.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646465427631.meta 2022-03-05 07:30:29,299 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646464820144.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646464820144.meta 2022-03-05 07:30:29,485 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 588 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:30:29,662 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646464822826 with entries=1300, filesize=576.91 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646465429299 2022-03-05 07:31:25,081 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2546 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:31:25,082 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1966 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:31:25,082 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 987 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:31:25,844 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 730 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:31:27,323 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1277 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:31:34,868 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 914 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:31:34,868 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 557 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:31:36,921 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 991 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=301.29 KB, freeSize=306.91 MB, max=307.20 MB, blockCount=9, accesses=5204, hits=5048, hitRatio=97.00%, , cachingAccesses=5064, cachingHits=5043, cachingHitsRatio=99.59%, evictions=1415, evicted=0, evictedPerRun=0.0 2022-03-05 07:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-05 07:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 07:31:48,641 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1278 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:31:48,641 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 705 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:31:58,376 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 662 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:31:58,376 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 662 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:32:34,614 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 836 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:36:32,590 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2152 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:36:32,590 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1977 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:36:32,590 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1821 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:36:32,590 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1681 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:36:32,590 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1425 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:36:37,314 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 838 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=301.29 KB, freeSize=306.91 MB, max=307.20 MB, blockCount=9, accesses=5232, hits=5076, hitRatio=97.02%, , cachingAccesses=5092, cachingHits=5071, cachingHitsRatio=99.59%, evictions=1445, evicted=0, evictedPerRun=0.0 2022-03-05 07:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 07:36:56,239 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df. because 80a1ffe1dc2d8f258a6367d204dc64df/e has an old edit so flush to free WALs after random delay 141944 ms 2022-03-05 07:37:06,502 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 888 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:37:08,352 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 600 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:37:21,089 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1190ms No GCs detected 2022-03-05 07:37:25,097 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3507ms No GCs detected 2022-03-05 07:37:31,867 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2300 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:37:31,867 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2210 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:37:31,867 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1432 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:37:31,867 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1158 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:37:31,867 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1131 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:37:36,269 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb. because 9b121c22e18472616debd0996ee2a4cb/e has an old edit so flush to free WALs after random delay 207158 ms 2022-03-05 07:37:53,023 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1059ms No GCs detected 2022-03-05 07:38:01,520 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 969 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:38:01,521 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 759 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:38:01,521 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 738 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:38:15,102 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2205 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:38:15,848 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 963 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:38:17,393 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2408 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:38:17,532 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1684 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:38:17,532 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1336 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:38:28,920 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1171 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:38:28,961 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1173 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:38:40,110 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2846ms No GCs detected 2022-03-05 07:38:47,417 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 07:38:47,510 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 07:38:47,510 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 07:38:47,510 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 07:38:47,510 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 07:38:47,422 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4568ms No GCs detected 2022-03-05 07:38:54,765 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:568) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.addBlock over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 07:38:54,772 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:568) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.addBlock over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 355ms. 2022-03-05 07:38:55,137 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: lientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:568) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.addBlock over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 2 failover attempts. Trying to failover after sleeping for 1391ms. 2022-03-05 07:38:56,532 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: lientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:568) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.addBlock over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 3 failover attempts. Trying to failover after sleeping for 2051ms. 2022-03-05 07:38:58,816 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646465429299 with entries=1061, filesize=465.35 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646465928178 2022-03-05 07:38:58,816 WARN [Close-WAL-Writer-36] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 07:38:58,816 WARN [Close-WAL-Writer-36] wal.AsyncProtobufLogWriter: normal close failed, try recover java.lang.IllegalStateException: should call flush first before calling close at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:510) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:514) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 07:38:58,816 INFO [Close-WAL-Writer-36] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646465429299 2022-03-05 07:38:58,821 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646465920127","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":18693,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":6,"class":"HRegionServer","multi.mutations":1} 2022-03-05 07:38:58,821 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646465927465","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":11355,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":45,"class":"HRegionServer","multi.mutations":1} 2022-03-05 07:38:58,821 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646465927465","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":11356,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":40,"class":"HRegionServer","multi.mutations":1} 2022-03-05 07:38:58,821 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646465927491","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":11330,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":24,"class":"HRegionServer","multi.mutations":1} 2022-03-05 07:38:58,821 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646465927465","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":11356,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":44,"class":"HRegionServer","multi.mutations":1} 2022-03-05 07:38:58,822 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646465920472","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":18350,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 07:38:58,822 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646465922617","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":16205,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":685,"class":"HRegionServer","multi.mutations":1} 2022-03-05 07:38:58,822 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646465920432","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":18390,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 07:38:58,822 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646465920110","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":18712,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 07:38:58,822 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646465925733","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":13089,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":3104,"class":"HRegionServer","multi.mutations":1} 2022-03-05 07:38:58,828 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646465928476","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":10351,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 07:38:58,828 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646465928318","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":10510,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":857,"class":"HRegionServer","multi.mutations":1} 2022-03-05 07:38:58,828 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646465927420","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":11407,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 07:38:58,837 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646465928323","responsesize":"98","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 14 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x06Kernel ","processingtimems":10514,"client":"172.30.41.118:54986","multi.service_calls":0,"queuetimems":5,"class":"HRegionServer","multi.mutations":14} 2022-03-05 07:38:58,839 INFO [Close-WAL-Writer-36] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646465429299 after 23ms 2022-03-05 07:39:04,149 INFO [Close-WAL-Writer-36] util.RecoverLeaseFSUtils: Recovered lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646465429299 after 5333ms 2022-03-05 07:39:18,183 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 80a1ffe1dc2d8f258a6367d204dc64df 1/1 column families, dataSize=299.81 KB heapSize=560.23 KB 2022-03-05 07:39:18,265 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=299.81 KB at sequenceid=5631 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/.tmp/e/d23ffedcc1e54724a51bc9b2b5e29b25 2022-03-05 07:39:18,272 INFO [MemStoreFlusher.0] retry.RetryInvocationHandler: doop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:433) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getBlockLocations over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 07:39:18,284 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/d23ffedcc1e54724a51bc9b2b5e29b25, entries=3552, sequenceid=5631, filesize=39.6 K 2022-03-05 07:39:18,285 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~299.81 KB/307007, heapSize ~560.21 KB/573656, currentSize=0 B/0 for 80a1ffe1dc2d8f258a6367d204dc64df in 102ms, sequenceid=5631, compaction requested=true 2022-03-05 07:39:18,285 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 80a1ffe1dc2d8f258a6367d204dc64df/e in aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df. 2022-03-05 07:39:18,286 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/c2aad83face044229391de152709b076, hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/d7dc25ebda75464d83d83c334bf4a5cb, hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/d23ffedcc1e54724a51bc9b2b5e29b25] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/.tmp, totalSize=218.5 K 2022-03-05 07:39:18,342 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 80a1ffe1dc2d8f258a6367d204dc64df#e#compaction#34 average throughput is 49.77 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 07:39:18,432 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 80a1ffe1dc2d8f258a6367d204dc64df/e of 80a1ffe1dc2d8f258a6367d204dc64df into 2acc4b57dd154eff9d74e3f65b375b2b(size=219.8 K), total size for store is 219.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 07:39:18,432 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., storeName=80a1ffe1dc2d8f258a6367d204dc64df/e, priority=13, startTime=1646465958285; duration=0sec 2022-03-05 07:39:25,810 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5. because 7534b3b1812808d2d3c16342fbf347a5/e has an old edit so flush to free WALs after random delay 105640 ms 2022-03-05 07:39:47,639 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 07:39:50,451 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5527 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:39:50,451 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5501 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:39:50,468 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4334 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:39:50,468 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3092 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:39:50,570 INFO [RS_COMPACTED_FILES_DISCHARGER-regionserver/cvp328:16201-6] retry.RetryInvocationHandler: 157) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:714) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.mkdirs over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 07:39:50,584 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 1 failover attempts. Trying to failover after sleeping for 502ms. 2022-03-05 07:39:51,086 WARN [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: A failover has occurred since the start of call #2593 ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 2022-03-05 07:39:56,088 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e. because 4333b489291b1f77671b8f2e78779a1e/__status_family___ has an old edit so flush to free WALs after random delay 286361 ms 2022-03-05 07:40:25,852 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. because 7d1b53259f1d8280a47f7704f98f2c9f/e has an old edit so flush to free WALs after random delay 141562 ms 2022-03-05 07:40:28,840 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646465427631.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646466028817.meta 2022-03-05 07:40:28,840 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646465427631.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646465427631.meta 2022-03-05 07:41:03,427 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 9b121c22e18472616debd0996ee2a4cb 1/1 column families, dataSize=592.79 KB heapSize=1.07 MB 2022-03-05 07:41:03,481 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=592.79 KB at sequenceid=7580 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/.tmp/e/ff4a55aa15ac4eeba02285987cdff15a 2022-03-05 07:41:03,497 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/ff4a55aa15ac4eeba02285987cdff15a, entries=6868, sequenceid=7580, filesize=72.3 K 2022-03-05 07:41:03,498 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~592.79 KB/607015, heapSize ~1.07 MB/1122624, currentSize=0 B/0 for 9b121c22e18472616debd0996ee2a4cb in 71ms, sequenceid=7580, compaction requested=true 2022-03-05 07:41:03,498 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 9b121c22e18472616debd0996ee2a4cb/e in aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb. 2022-03-05 07:41:03,499 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/c3d0c108c9db4da9a93a2dd4e7120638, hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/ce9d47626e08440181f7ae5ae1389f92, hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/ff4a55aa15ac4eeba02285987cdff15a] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/.tmp, totalSize=337.3 K 2022-03-05 07:41:03,574 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 9b121c22e18472616debd0996ee2a4cb#e#compaction#36 average throughput is 49.15 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 07:41:03,642 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 9b121c22e18472616debd0996ee2a4cb/e of 9b121c22e18472616debd0996ee2a4cb into 8f6a51bacef345debad6a0e6ceb856e5(size=359.0 K), total size for store is 359.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 07:41:03,642 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., storeName=9b121c22e18472616debd0996ee2a4cb/e, priority=13, startTime=1646466063498; duration=0sec 2022-03-05 07:41:11,451 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 7534b3b1812808d2d3c16342fbf347a5 1/1 column families, dataSize=178.47 KB heapSize=331.80 KB 2022-03-05 07:41:11,511 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=178.47 KB at sequenceid=4490 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/.tmp/e/6e679a9ac8674743b97bc26b8e181811 2022-03-05 07:41:11,529 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/6e679a9ac8674743b97bc26b8e181811, entries=2061, sequenceid=4490, filesize=24.8 K 2022-03-05 07:41:11,530 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~178.47 KB/182753, heapSize ~331.78 KB/339744, currentSize=0 B/0 for 7534b3b1812808d2d3c16342fbf347a5 in 80ms, sequenceid=4490, compaction requested=false 2022-03-05 07:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=301.29 KB, freeSize=306.91 MB, max=307.20 MB, blockCount=9, accesses=5322, hits=5098, hitRatio=95.79%, , cachingAccesses=5114, cachingHits=5093, cachingHitsRatio=99.59%, evictions=1475, evicted=0, evictedPerRun=0.0 2022-03-05 07:41:45,828 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb. because ccdacf4b414cb8d1bc548efc77c663cb/e has an old edit so flush to free WALs after random delay 195070 ms 2022-03-05 07:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 07:42:47,414 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 7d1b53259f1d8280a47f7704f98f2c9f 1/1 column families, dataSize=665.96 KB heapSize=1.12 MB 2022-03-05 07:42:47,473 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=665.96 KB at sequenceid=5573 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/.tmp/e/a54f71c0fd05425592737d444beec270 2022-03-05 07:42:47,488 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/a54f71c0fd05425592737d444beec270, entries=6438, sequenceid=5573, filesize=64.2 K 2022-03-05 07:42:47,489 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~665.96 KB/681947, heapSize ~1.12 MB/1170952, currentSize=0 B/0 for 7d1b53259f1d8280a47f7704f98f2c9f in 75ms, sequenceid=5573, compaction requested=true 2022-03-05 07:42:47,490 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 7d1b53259f1d8280a47f7704f98f2c9f/e in aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. 2022-03-05 07:42:47,490 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/6f863164283141d9a111eeda1fc65a11, hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/5dfc562018a54a118fecf49a66576877, hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/a54f71c0fd05425592737d444beec270] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/.tmp, totalSize=325.5 K 2022-03-05 07:42:47,559 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 7d1b53259f1d8280a47f7704f98f2c9f#e#compaction#39 average throughput is 54.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 07:42:47,627 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 7d1b53259f1d8280a47f7704f98f2c9f/e of 7d1b53259f1d8280a47f7704f98f2c9f into 12e47a2953e243249e8100a6f89b5472(size=366.9 K), total size for store is 366.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 07:42:47,627 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., storeName=7d1b53259f1d8280a47f7704f98f2c9f/e, priority=13, startTime=1646466167489; duration=0sec 2022-03-05 07:44:42,449 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 4333b489291b1f77671b8f2e78779a1e 1/1 column families, dataSize=8.12 KB heapSize=12.96 KB 2022-03-05 07:44:42,497 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=8.12 KB at sequenceid=328 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/.tmp/__status_family___/5bcaebb17c254429af8200d0c4ef143c 2022-03-05 07:44:42,509 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/5bcaebb17c254429af8200d0c4ef143c, entries=3, sequenceid=328, filesize=5.1 K 2022-03-05 07:44:42,510 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~8.12 KB/8313, heapSize ~12.95 KB/13256, currentSize=0 B/0 for 4333b489291b1f77671b8f2e78779a1e in 61ms, sequenceid=328, compaction requested=false 2022-03-05 07:45:00,899 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing ccdacf4b414cb8d1bc548efc77c663cb 1/1 column families, dataSize=293.31 KB heapSize=506.95 KB 2022-03-05 07:45:00,977 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=293.31 KB at sequenceid=3114 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/.tmp/e/ed4c2412108b498492abfbfdf683bf31 2022-03-05 07:45:00,993 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/ed4c2412108b498492abfbfdf683bf31, entries=2899, sequenceid=3114, filesize=30.4 K 2022-03-05 07:45:00,994 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~293.31 KB/300352, heapSize ~506.94 KB/519104, currentSize=0 B/0 for ccdacf4b414cb8d1bc548efc77c663cb in 96ms, sequenceid=3114, compaction requested=true 2022-03-05 07:45:00,995 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of ccdacf4b414cb8d1bc548efc77c663cb/e in aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb. 2022-03-05 07:45:00,995 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/8719de3605884dc2a02d513f519243b3, hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/fcdb7dada2f04aeba960d89d30050603, hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/ed4c2412108b498492abfbfdf683bf31] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/.tmp, totalSize=191.8 K 2022-03-05 07:45:01,039 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: ccdacf4b414cb8d1bc548efc77c663cb#e#compaction#42 average throughput is 58.63 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 07:45:01,113 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in ccdacf4b414cb8d1bc548efc77c663cb/e of ccdacf4b414cb8d1bc548efc77c663cb into 5f6d49dcd992478ca0cb9e16621ac5d7(size=202.5 K), total size for store is 202.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 07:45:01,113 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., storeName=ccdacf4b414cb8d1bc548efc77c663cb/e, priority=13, startTime=1646466300995; duration=0sec 2022-03-05 07:46:04,385 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1026 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:46:06,271 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391. because 0a764e51258807790269e46c55b99391/e has an old edit so flush to free WALs after random delay 264763 ms 2022-03-05 07:46:08,322 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 513 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:46:13,465 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 619 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 07:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=301.29 KB, freeSize=306.91 MB, max=307.20 MB, blockCount=9, accesses=5417, hits=5120, hitRatio=94.52%, , cachingAccesses=5136, cachingHits=5115, cachingHitsRatio=99.59%, evictions=1505, evicted=0, evictedPerRun=0.0 2022-03-05 07:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 07:48:48,887 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646465928178 with entries=1339, filesize=591.28 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646466528855 2022-03-05 07:48:48,888 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646461987115 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646461987115 2022-03-05 07:48:48,900 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646462355575 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646462355575 2022-03-05 07:48:48,909 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646462386427 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646462386427 2022-03-05 07:50:28,923 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646466028817.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646466628889.meta 2022-03-05 07:50:28,923 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646466028817.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646466028817.meta 2022-03-05 07:50:31,034 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 0a764e51258807790269e46c55b99391 1/1 column families, dataSize=964.39 KB heapSize=1.69 MB 2022-03-05 07:50:32,062 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=964.39 KB at sequenceid=10009 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/.tmp/e/a1407900c08848dabd9c9a80d466c296 2022-03-05 07:50:32,079 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/a1407900c08848dabd9c9a80d466c296, entries=10361, sequenceid=10009, filesize=93.9 K 2022-03-05 07:50:32,080 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~964.39 KB/987534, heapSize ~1.69 MB/1773784, currentSize=207 B/207 for 0a764e51258807790269e46c55b99391 in 1046ms, sequenceid=10009, compaction requested=true 2022-03-05 07:50:32,081 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 0a764e51258807790269e46c55b99391/e in aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391. 2022-03-05 07:50:32,081 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/e741166706934b59a37722f54787b6b3, hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/0333a2bb2b2147d192b32ccff4a351ca, hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/a1407900c08848dabd9c9a80d466c296] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/.tmp, totalSize=447.8 K 2022-03-05 07:50:32,206 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 0a764e51258807790269e46c55b99391#e#compaction#44 average throughput is 37.83 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 07:50:32,290 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 0a764e51258807790269e46c55b99391/e of 0a764e51258807790269e46c55b99391 into e0866d8a50c74b098dfedf756fb9610e(size=510.3 K), total size for store is 510.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 07:50:32,290 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391., storeName=0a764e51258807790269e46c55b99391/e, priority=13, startTime=1646466632081; duration=0sec 2022-03-05 07:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=301.29 KB, freeSize=306.91 MB, max=307.20 MB, blockCount=9, accesses=5508, hits=5143, hitRatio=93.37%, , cachingAccesses=5158, cachingHits=5137, cachingHitsRatio=99.59%, evictions=1535, evicted=0, evictedPerRun=0.0 2022-03-05 07:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 07:52:36,211 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 626 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:55:30,718 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2970 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:55:30,718 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2538 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:55:30,718 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2209 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:55:30,718 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1863 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:55:30,719 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1077 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:55:43,722 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1529ms No GCs detected 2022-03-05 07:55:56,303 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 07:56:01,826 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1095 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:56:01,827 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 661 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:56:08,487 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 704 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:56:18,147 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 735 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=5530, hits=5165, hitRatio=93.40%, , cachingAccesses=5180, cachingHits=5159, cachingHitsRatio=99.59%, evictions=1565, evicted=0, evictedPerRun=0.0 2022-03-05 07:56:47,146 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 937 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 07:56:58,955 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1012 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:56:58,955 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 842 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 07:58:49,265 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646466528855 with entries=1285, filesize=566.62 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646467129246 2022-03-05 07:58:49,265 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646462605225 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646462605225 2022-03-05 07:58:49,270 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646462779491 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646462779491 2022-03-05 07:58:49,275 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646462950778 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646462950778 2022-03-05 07:58:49,280 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646462974968 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646462974968 2022-03-05 07:58:49,283 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646463404518 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646463404518 2022-03-05 07:58:49,286 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646464005228 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646464005228 2022-03-05 07:58:49,292 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646464272855 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646464272855 2022-03-05 07:58:49,295 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646464822826 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646464822826 2022-03-05 07:58:49,299 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646465429299 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646465429299 2022-03-05 07:59:16,707 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1722 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:00:07,273 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1249 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:00:12,930 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1635 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:00:12,930 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1228 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:00:12,931 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 978 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:00:29,283 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646466628889.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646467229267.meta 2022-03-05 08:00:29,283 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646466628889.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646466628889.meta 2022-03-05 08:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=5552, hits=5187, hitRatio=93.43%, , cachingAccesses=5202, cachingHits=5181, cachingHitsRatio=99.60%, evictions=1595, evicted=0, evictedPerRun=0.0 2022-03-05 08:01:44,645 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1723 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 08:01:58,802 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1441 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:01:58,802 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1078 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:01:58,802 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1078 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:01:58,802 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 867 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:01:58,802 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 667 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:02:05,772 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2019 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:02:06,808 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1038 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:02:06,808 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1032 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:02:32,264 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5844 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:02:32,264 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5649 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:02:33,040 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4445 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:02:33,040 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4002 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:02:33,040 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3383 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:02:33,040 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2931 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:02:33,040 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2915 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:02:33,040 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2531 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:02:33,040 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1634 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:02:33,040 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 790 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:02:33,040 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 776 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:02:33,040 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 741 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:02:34,945 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1904 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:02:34,945 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1903 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:02:34,945 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1731 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:02:34,945 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1731 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:02:34,945 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1731 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:02:34,945 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1468 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:02:34,946 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 747 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:03:04,955 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 776 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:03:04,955 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 547 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:03:16,579 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 955 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:03:17,892 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1312 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=5574, hits=5209, hitRatio=93.45%, , cachingAccesses=5224, cachingHits=5203, cachingHitsRatio=99.60%, evictions=1625, evicted=0, evictedPerRun=0.0 2022-03-05 08:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 08:07:05,657 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1865 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:07:05,657 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1470 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:07:17,865 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1916 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:07:23,732 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 952 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:07:23,732 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 711 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:07:27,356 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1040 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:07:30,425 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1196ms No GCs detected 2022-03-05 08:07:30,426 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3783 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:08:00,980 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 08:08:50,529 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646467129246 with entries=1287, filesize=571.76 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646467730508 2022-03-05 08:10:32,570 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1471 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:10:32,571 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1266 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:10:32,571 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1126 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:10:32,571 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1115 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:10:32,571 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1000 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:10:32,891 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646467229267.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646467830531.meta 2022-03-05 08:10:32,894 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646467229267.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646467229267.meta 2022-03-05 08:10:41,415 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 936 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:10:43,011 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1248 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:10:43,011 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1120 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:10:47,012 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 756 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:10:52,911 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 723 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:11:06,694 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1088 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:11:29,751 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1194 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:11:29,751 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1194 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=5596, hits=5231, hitRatio=93.48%, , cachingAccesses=5246, cachingHits=5225, cachingHitsRatio=99.60%, evictions=1655, evicted=0, evictedPerRun=0.0 2022-03-05 08:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 08:12:02,993 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 838 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:12:08,885 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 620 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:12:50,463 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1830ms No GCs detected 2022-03-05 08:13:01,674 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 924 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:13:01,674 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 572 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:13:11,000 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 545 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:13:32,834 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 610 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:13:34,316 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 505 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:13:35,057 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 862 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:16:25,825 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 877 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=5618, hits=5253, hitRatio=93.50%, , cachingAccesses=5268, cachingHits=5247, cachingHitsRatio=99.60%, evictions=1685, evicted=0, evictedPerRun=0.0 2022-03-05 08:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 08:17:23,022 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 545 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:17:41,594 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 891 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:17:46,474 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4874 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:17:46,474 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4780 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:17:46,474 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4573 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:17:46,475 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2003 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:17:46,475 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1648 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:17:46,475 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1522 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:17:58,571 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 719 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:17:58,571 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 517 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:18:00,858 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 587 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:18:17,985 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1763 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:18:19,712 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1726 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:18:31,715 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2054 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:18:31,715 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1589 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:18:31,715 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1290 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:18:31,715 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1151 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:18:31,715 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 992 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:18:31,715 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 620 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:18:43,671 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1754ms No GCs detected 2022-03-05 08:18:53,373 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646467730508 with entries=1300, filesize=572 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646468333349 2022-03-05 08:19:53,673 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 598 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:19:53,673 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 506 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:20:00,879 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1331 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:20:00,879 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1331 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:20:00,879 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1215 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:20:00,879 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 740 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:20:00,879 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 500 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:20:03,905 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2810 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:20:03,905 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1906 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:20:03,905 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1119 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:20:26,571 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1678 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:20:26,572 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1415 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:20:26,572 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 596 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:20:33,893 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646467830531.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646468433864.meta 2022-03-05 08:20:33,893 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646467830531.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646467830531.meta 2022-03-05 08:21:01,600 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4002 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:21:01,600 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4001 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:21:01,600 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3665 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:21:01,659 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3172 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:21:01,659 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3111 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:21:01,659 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2175 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:21:01,659 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1941 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:21:08,836 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1132 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:21:12,021 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1564 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:21:12,021 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1532 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=5640, hits=5275, hitRatio=93.53%, , cachingAccesses=5290, cachingHits=5269, cachingHitsRatio=99.60%, evictions=1715, evicted=0, evictedPerRun=0.0 2022-03-05 08:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 08:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=5662, hits=5297, hitRatio=93.55%, , cachingAccesses=5312, cachingHits=5291, cachingHitsRatio=99.60%, evictions=1745, evicted=0, evictedPerRun=0.0 2022-03-05 08:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 08:28:53,919 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646468333349 with entries=1297, filesize=566.81 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646468933900 2022-03-05 08:29:03,581 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1423 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 08:29:03,581 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1423 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 08:29:03,581 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 793 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 08:29:03,581 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 591 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 08:30:34,154 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646468433864.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646469034079.meta 2022-03-05 08:30:34,155 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646468433864.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646468433864.meta 2022-03-05 08:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=5684, hits=5319, hitRatio=93.58%, , cachingAccesses=5334, cachingHits=5313, cachingHitsRatio=99.61%, evictions=1775, evicted=0, evictedPerRun=0.0 2022-03-05 08:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-05 08:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 08:33:24,107 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6678 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 08:33:24,107 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6237 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 08:33:28,621 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2300ms No GCs detected 2022-03-05 08:33:31,056 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 8861 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 08:33:32,415 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3293ms No GCs detected 2022-03-05 08:33:32,398 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646469202194","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":10203,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 08:33:32,531 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 9212 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 08:33:32,987 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 8718 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 08:33:32,987 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1931 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 08:33:42,112 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1577ms No GCs detected 2022-03-05 08:33:42,114 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3904 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 08:33:43,062 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 08:33:48,048 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 593 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 08:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=5712, hits=5347, hitRatio=93.61%, , cachingAccesses=5362, cachingHits=5341, cachingHitsRatio=99.61%, evictions=1805, evicted=0, evictedPerRun=0.0 2022-03-05 08:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 08:38:12,040 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3330 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 08:38:54,736 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646468933900 with entries=1304, filesize=570.77 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646469534720 2022-03-05 08:39:36,134 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df. because 80a1ffe1dc2d8f258a6367d204dc64df/e has an old edit so flush to free WALs after random delay 275914 ms 2022-03-05 08:40:34,814 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646469034079.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646469634742.meta 2022-03-05 08:40:34,814 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646469034079.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646469034079.meta 2022-03-05 08:41:05,796 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb. because 9b121c22e18472616debd0996ee2a4cb/e has an old edit so flush to free WALs after random delay 181860 ms 2022-03-05 08:41:16,503 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5. because 7534b3b1812808d2d3c16342fbf347a5/e has an old edit so flush to free WALs after random delay 106244 ms 2022-03-05 08:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=5734, hits=5369, hitRatio=93.63%, , cachingAccesses=5384, cachingHits=5363, cachingHitsRatio=99.61%, evictions=1835, evicted=0, evictedPerRun=0.0 2022-03-05 08:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 08:42:56,425 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. because 7d1b53259f1d8280a47f7704f98f2c9f/e has an old edit so flush to free WALs after random delay 86311 ms 2022-03-05 08:43:02,747 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 7534b3b1812808d2d3c16342fbf347a5 1/1 column families, dataSize=181.36 KB heapSize=338.03 KB 2022-03-05 08:43:02,799 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=181.36 KB at sequenceid=5468 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/.tmp/e/c59750ef825842c891c4d8aa8bf0db64 2022-03-05 08:43:02,806 INFO [MemStoreFlusher.1] retry.RetryInvocationHandler: doop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:433) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getBlockLocations over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 08:43:02,813 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/c59750ef825842c891c4d8aa8bf0db64, entries=2110, sequenceid=5468, filesize=25.3 K 2022-03-05 08:43:02,814 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~181.36 KB/185710, heapSize ~338.02 KB/346128, currentSize=0 B/0 for 7534b3b1812808d2d3c16342fbf347a5 in 67ms, sequenceid=5468, compaction requested=true 2022-03-05 08:43:02,815 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 7534b3b1812808d2d3c16342fbf347a5/e in aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5. 2022-03-05 08:43:02,815 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/26b6ff8862b947f38dd508b795e21c3d, hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/6e679a9ac8674743b97bc26b8e181811, hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/c59750ef825842c891c4d8aa8bf0db64] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/.tmp, totalSize=162.7 K 2022-03-05 08:43:02,852 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 7534b3b1812808d2d3c16342fbf347a5#e#compaction#46 average throughput is 49.26 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 08:43:02,887 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 7534b3b1812808d2d3c16342fbf347a5/e of 7534b3b1812808d2d3c16342fbf347a5 into dde113ca411d4939b7fcfe2f368dda2c(size=155.2 K), total size for store is 155.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 08:43:02,887 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., storeName=7534b3b1812808d2d3c16342fbf347a5/e, priority=13, startTime=1646469782815; duration=0sec 2022-03-05 08:44:07,656 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 9b121c22e18472616debd0996ee2a4cb 1/1 column families, dataSize=586.68 KB heapSize=1.06 MB 2022-03-05 08:44:07,707 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=586.68 KB at sequenceid=9316 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/.tmp/e/25b4aad0d2bb4e18b44232186316eaa5 2022-03-05 08:44:07,720 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/25b4aad0d2bb4e18b44232186316eaa5, entries=6773, sequenceid=9316, filesize=71.8 K 2022-03-05 08:44:07,721 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~586.68 KB/600759, heapSize ~1.06 MB/1108888, currentSize=0 B/0 for 9b121c22e18472616debd0996ee2a4cb in 65ms, sequenceid=9316, compaction requested=false 2022-03-05 08:44:12,048 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 80a1ffe1dc2d8f258a6367d204dc64df 1/1 column families, dataSize=307.46 KB heapSize=573.18 KB 2022-03-05 08:44:12,160 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=307.46 KB at sequenceid=6876 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/.tmp/e/d96edbddb0c44e53a3ecb754c1677800 2022-03-05 08:44:12,171 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/d96edbddb0c44e53a3ecb754c1677800, entries=3627, sequenceid=6876, filesize=40.3 K 2022-03-05 08:44:12,172 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~307.46 KB/314834, heapSize ~573.16 KB/586920, currentSize=0 B/0 for 80a1ffe1dc2d8f258a6367d204dc64df in 124ms, sequenceid=6876, compaction requested=false 2022-03-05 08:44:22,736 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 7d1b53259f1d8280a47f7704f98f2c9f 1/1 column families, dataSize=658.67 KB heapSize=1.10 MB 2022-03-05 08:44:22,782 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=658.67 KB at sequenceid=6658 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/.tmp/e/78d1b77deeb747c0a9bd912c715f15d3 2022-03-05 08:44:22,792 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/78d1b77deeb747c0a9bd912c715f15d3, entries=6373, sequenceid=6658, filesize=63.8 K 2022-03-05 08:44:22,793 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~658.67 KB/674482, heapSize ~1.10 MB/1158640, currentSize=0 B/0 for 7d1b53259f1d8280a47f7704f98f2c9f in 57ms, sequenceid=6658, compaction requested=false 2022-03-05 08:44:44,526 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1667 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 08:45:06,693 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb. because ccdacf4b414cb8d1bc548efc77c663cb/e has an old edit so flush to free WALs after random delay 235236 ms 2022-03-05 08:45:09,220 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 975 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 08:45:12,525 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 601 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 08:45:15,057 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 08:45:23,884 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1524ms No GCs detected 2022-03-05 08:45:24,422 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2136 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 08:45:24,422 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 534 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 08:45:35,796 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e. because 4333b489291b1f77671b8f2e78779a1e/__status_family___ has an old edit so flush to free WALs after random delay 119106 ms 2022-03-05 08:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=5775, hits=5391, hitRatio=93.35%, , cachingAccesses=5406, cachingHits=5385, cachingHitsRatio=99.61%, evictions=1865, evicted=0, evictedPerRun=0.0 2022-03-05 08:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 08:47:34,905 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 4333b489291b1f77671b8f2e78779a1e 1/1 column families, dataSize=7.75 KB heapSize=12.36 KB 2022-03-05 08:47:35,241 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=7.75 KB at sequenceid=393 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/.tmp/__status_family___/360a11db492b44d28bb119e4deb43797 2022-03-05 08:47:35,271 INFO [MemStoreFlusher.1] retry.RetryInvocationHandler: doop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:433) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getBlockLocations over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 08:47:35,339 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/360a11db492b44d28bb119e4deb43797, entries=3, sequenceid=393, filesize=5.1 K 2022-03-05 08:47:35,357 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~7.75 KB/7931, heapSize ~12.34 KB/12640, currentSize=0 B/0 for 4333b489291b1f77671b8f2e78779a1e in 452ms, sequenceid=393, compaction requested=true 2022-03-05 08:47:35,357 INFO [regionserver/cvp328:16201-longCompactions-0] regionserver.HRegion: Starting compaction of 4333b489291b1f77671b8f2e78779a1e/__status_family___ in __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e. 2022-03-05 08:47:35,358 INFO [regionserver/cvp328:16201-longCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/bb47857b71e44063ae29d154cb9b32a9, hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/5bcaebb17c254429af8200d0c4ef143c, hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/360a11db492b44d28bb119e4deb43797] into tmpdir=hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/.tmp, totalSize=15.4 K 2022-03-05 08:47:35,486 INFO [regionserver/cvp328:16201-longCompactions-0] throttle.PressureAwareThroughputController: 4333b489291b1f77671b8f2e78779a1e#__status_family___#compaction#51 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 08:47:35,741 INFO [regionserver/cvp328:16201-longCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 4333b489291b1f77671b8f2e78779a1e/__status_family___ of 4333b489291b1f77671b8f2e78779a1e into f76c2bd644a34f6284d6c442ed66fbaf(size=5.2 K), total size for store is 5.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 08:47:35,741 INFO [regionserver/cvp328:16201-longCompactions-0] regionserver.CompactSplit: Completed compaction region=__test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., storeName=4333b489291b1f77671b8f2e78779a1e/__status_family___, priority=13, startTime=1646470055357; duration=0sec 2022-03-05 08:48:14,136 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1277 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 08:48:55,338 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646469534720 with entries=1312, filesize=576.45 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646470135323 2022-03-05 08:49:01,930 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing ccdacf4b414cb8d1bc548efc77c663cb 1/1 column families, dataSize=295.44 KB heapSize=511.52 KB 2022-03-05 08:49:02,925 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 995 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:49:02,926 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 658 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:49:03,963 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1016 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:49:04,241 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=295.44 KB at sequenceid=3777 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/.tmp/e/2ac85f3be0b243d9b9a0eb3277c1460d 2022-03-05 08:49:04,261 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/2ac85f3be0b243d9b9a0eb3277c1460d, entries=2932, sequenceid=3777, filesize=30.8 K 2022-03-05 08:49:04,262 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~295.44 KB/302526, heapSize ~511.50 KB/523776, currentSize=215 B/215 for ccdacf4b414cb8d1bc548efc77c663cb in 2332ms, sequenceid=3777, compaction requested=false 2022-03-05 08:50:35,410 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646469634742.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646470235341.meta 2022-03-05 08:50:35,419 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646469634742.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646469634742.meta 2022-03-05 08:50:36,112 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391. because 0a764e51258807790269e46c55b99391/e has an old edit so flush to free WALs after random delay 177743 ms 2022-03-05 08:51:18,046 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 08:51:19,954 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 351ms. 2022-03-05 08:51:21,260 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5902 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:51:21,260 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4713 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:51:21,260 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4679 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:51:21,260 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3214 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:51:21,260 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2368 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:51:23,404 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2143 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:51:23,404 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1042 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=5800, hits=5413, hitRatio=93.33%, , cachingAccesses=5428, cachingHits=5407, cachingHitsRatio=99.61%, evictions=1895, evicted=0, evictedPerRun=0.0 2022-03-05 08:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 08:51:55,209 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1749 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:52:02,327 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 935 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:52:02,327 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 692 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:52:05,804 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1642 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:52:08,382 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1074ms No GCs detected 2022-03-05 08:52:09,171 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3522 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:52:09,917 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3108 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:52:09,917 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1492 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:52:11,176 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1259 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:52:11,176 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1258 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:52:11,176 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 716 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:52:19,229 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1159ms No GCs detected 2022-03-05 08:52:19,280 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1345 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 08:53:33,855 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 0a764e51258807790269e46c55b99391 1/1 column families, dataSize=945.01 KB heapSize=1.66 MB 2022-03-05 08:53:34,215 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=945.01 KB at sequenceid=12426 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/.tmp/e/329674adf6d54e4a9f69ea5368d06bf1 2022-03-05 08:53:34,391 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/329674adf6d54e4a9f69ea5368d06bf1, entries=10199, sequenceid=12426, filesize=94.1 K 2022-03-05 08:53:34,412 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~945.01 KB/967693, heapSize ~1.66 MB/1741392, currentSize=201 B/201 for 0a764e51258807790269e46c55b99391 in 557ms, sequenceid=12426, compaction requested=false 2022-03-05 08:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=5822, hits=5435, hitRatio=93.35%, , cachingAccesses=5450, cachingHits=5429, cachingHitsRatio=99.61%, evictions=1924, evicted=0, evictedPerRun=0.0 2022-03-05 08:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 08:58:55,817 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646470135323 with entries=1302, filesize=562.06 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646470735728 2022-03-05 08:58:55,817 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646465928178 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646465928178 2022-03-05 08:58:55,823 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646466528855 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646466528855 2022-03-05 08:58:55,829 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646467129246 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646467129246 2022-03-05 08:58:55,833 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646467730508 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646467730508 2022-03-05 08:58:55,839 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646468333349 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646468333349 2022-03-05 08:58:55,844 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646468933900 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646468933900 2022-03-05 09:00:35,856 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646470235341.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646470835819.meta 2022-03-05 09:00:35,856 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646470235341.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646470235341.meta 2022-03-05 09:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=5844, hits=5457, hitRatio=93.38%, , cachingAccesses=5472, cachingHits=5451, cachingHitsRatio=99.62%, evictions=1954, evicted=0, evictedPerRun=0.0 2022-03-05 09:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 09:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=5866, hits=5479, hitRatio=93.40%, , cachingAccesses=5494, cachingHits=5473, cachingHitsRatio=99.62%, evictions=1984, evicted=0, evictedPerRun=0.0 2022-03-05 09:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 09:08:56,579 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646470735728 with entries=1295, filesize=570.53 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646471336564 2022-03-05 09:10:36,604 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646470835819.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646471436580.meta 2022-03-05 09:10:36,604 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646470835819.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646470835819.meta 2022-03-05 09:10:58,663 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 528 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 09:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=5888, hits=5501, hitRatio=93.43%, , cachingAccesses=5516, cachingHits=5495, cachingHitsRatio=99.62%, evictions=2014, evicted=0, evictedPerRun=0.0 2022-03-05 09:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 09:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=5910, hits=5523, hitRatio=93.45%, , cachingAccesses=5538, cachingHits=5517, cachingHitsRatio=99.62%, evictions=2044, evicted=0, evictedPerRun=0.0 2022-03-05 09:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 09:18:01,862 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 951 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 09:18:01,862 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 935 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 09:18:01,863 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 791 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 09:18:01,863 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 789 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 09:18:01,863 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 752 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 09:18:01,863 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 573 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 09:18:56,679 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646471336564 with entries=1291, filesize=572.59 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646471936613 2022-03-05 09:20:36,696 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646471436580.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646472036683.meta 2022-03-05 09:20:36,696 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646471436580.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646471436580.meta 2022-03-05 09:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=5932, hits=5545, hitRatio=93.48%, , cachingAccesses=5560, cachingHits=5539, cachingHitsRatio=99.62%, evictions=2074, evicted=0, evictedPerRun=0.0 2022-03-05 09:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 09:24:05,889 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 920 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 09:24:10,289 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2340 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 09:24:11,299 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2829 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 09:24:11,299 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1810 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 09:25:42,414 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 585 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 09:25:42,414 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 504 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 09:26:02,458 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3733ms No GCs detected 2022-03-05 09:26:02,489 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 09:26:02,607 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 09:26:02,616 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 09:26:02,616 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 09:26:02,616 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 09:26:02,616 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.sync(AsyncProtobufLogWriter.java:148) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:394) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.consume(AsyncFSWAL.java:581) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 09:26:02,776 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646471936613 with entries=905, filesize=393.43 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646472362610 2022-03-05 09:26:02,776 WARN [Close-WAL-Writer-47] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 09:26:02,776 WARN [Close-WAL-Writer-47] wal.AsyncProtobufLogWriter: normal close failed, try recover java.lang.IllegalStateException: should call flush first before calling close at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:510) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:514) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 09:26:02,777 INFO [Close-WAL-Writer-47] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646471936613 2022-03-05 09:26:02,819 INFO [Close-WAL-Writer-47] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646471936613 after 42ms 2022-03-05 09:26:06,820 INFO [Close-WAL-Writer-47] retry.RetryInvocationHandler: .apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.recoverLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 09:26:06,827 INFO [Close-WAL-Writer-47] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646471936613 after 4050ms 2022-03-05 09:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=5954, hits=5567, hitRatio=93.50%, , cachingAccesses=5582, cachingHits=5561, cachingHitsRatio=99.62%, evictions=2104, evicted=0, evictedPerRun=0.0 2022-03-05 09:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 09:28:39,726 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 566 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 09:28:41,670 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1201 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 09:28:44,709 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1907ms No GCs detected 2022-03-05 09:28:44,710 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3152 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 09:28:44,904 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2996 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 09:28:46,485 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 593 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 09:28:48,361 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2044 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 09:28:48,361 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1875 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 09:29:11,824 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1344ms No GCs detected 2022-03-05 09:29:13,045 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1220 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 09:29:13,045 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1147 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 09:29:13,045 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1017 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 09:29:58,965 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1456 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 09:29:58,965 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 926 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 09:30:44,726 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646472036683.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646472644709.meta 2022-03-05 09:30:44,726 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646472036683.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646472036683.meta 2022-03-05 09:31:01,098 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 721 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 09:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=5976, hits=5589, hitRatio=93.52%, , cachingAccesses=5604, cachingHits=5583, cachingHitsRatio=99.63%, evictions=2134, evicted=0, evictedPerRun=0.0 2022-03-05 09:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-05 09:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 09:32:33,658 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 829 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 09:32:33,658 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 694 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 09:33:27,942 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1723ms No GCs detected 2022-03-05 09:34:09,350 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4711ms No GCs detected 2022-03-05 09:34:14,659 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1725ms No GCs detected 2022-03-05 09:36:08,000 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646472362610 with entries=1322, filesize=580.50 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646472967986 2022-03-05 09:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=6004, hits=5617, hitRatio=93.55%, , cachingAccesses=5632, cachingHits=5611, cachingHitsRatio=99.63%, evictions=2164, evicted=0, evictedPerRun=0.0 2022-03-05 09:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 09:37:21,250 WARN [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 15506ms No GCs detected 2022-03-05 09:37:23,891 INFO [regionserver/cvp328:16201.Chore.1] hbase.ScheduledChore: Chore: MemstoreFlusherChore missed its start time 2022-03-05 09:37:23,891 INFO [regionserver/cvp328:16201.Chore.1] hbase.ScheduledChore: Chore: CompactionChecker missed its start time 2022-03-05 09:37:23,904 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.sync(AsyncProtobufLogWriter.java:148) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:394) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.consume(AsyncFSWAL.java:581) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 09:37:23,931 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2180ms No GCs detected 2022-03-05 09:37:24,191 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646472967986 with entries=142, filesize=70.03 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646473043904 2022-03-05 09:37:24,192 WARN [Close-WAL-Writer-49] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 09:37:24,200 WARN [Close-WAL-Writer-49] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 09:37:24,201 INFO [Close-WAL-Writer-49] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646472967986 2022-03-05 09:37:24,356 INFO [Close-WAL-Writer-49] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646472967986 after 155ms 2022-03-05 09:37:28,358 INFO [Close-WAL-Writer-49] retry.RetryInvocationHandler: .apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.recoverLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 09:37:29,185 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1101 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 09:37:29,185 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 998 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 09:37:29,185 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 663 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 09:37:29,537 INFO [Close-WAL-Writer-49] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646472967986 after 5336ms 2022-03-05 09:37:34,438 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646472644709.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646473054195.meta 2022-03-05 09:37:34,438 WARN [Close-WAL-Writer-50] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 09:37:34,438 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646472644709.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646472644709.meta 2022-03-05 09:37:34,438 WARN [Close-WAL-Writer-50] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 09:37:34,438 INFO [Close-WAL-Writer-50] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646472644709.meta 2022-03-05 09:37:34,456 INFO [Close-WAL-Writer-50] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646472644709.meta after 18ms 2022-03-05 09:37:38,458 WARN [Close-WAL-Writer-50] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646472644709.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646472644709.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 09:41:26,130 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 09:41:26,132 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 09:41:26,132 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 09:41:26,132 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 09:41:26,132 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 09:41:26,133 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 09:41:26,133 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 09:41:27,772 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646473043904 with entries=544, filesize=247.75 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646473286132 2022-03-05 09:41:27,772 WARN [Close-WAL-Writer-50] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 09:41:27,772 WARN [Close-WAL-Writer-50] wal.AsyncProtobufLogWriter: normal close failed, try recover java.lang.IllegalStateException: should call flush first before calling close at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:510) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:514) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 09:41:27,772 INFO [Close-WAL-Writer-50] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646473043904 2022-03-05 09:41:27,774 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646473277431","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":10343,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 09:41:27,775 INFO [Close-WAL-Writer-50] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646473043904 after 3ms 2022-03-05 09:41:31,776 INFO [Close-WAL-Writer-50] util.RecoverLeaseFSUtils: Recovered lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646473043904 after 4004ms 2022-03-05 09:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=333.17 KB, freeSize=306.87 MB, max=307.20 MB, blockCount=20, accesses=6126, hits=5726, hitRatio=93.47%, , cachingAccesses=5754, cachingHits=5720, cachingHitsRatio=99.41%, evictions=2192, evicted=0, evictedPerRun=0.0 2022-03-05 09:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 09:43:23,938 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5. because 7534b3b1812808d2d3c16342fbf347a5/e has an old edit so flush to free WALs after random delay 264922 ms 2022-03-05 09:44:14,364 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb. because 9b121c22e18472616debd0996ee2a4cb/e has an old edit so flush to free WALs after random delay 22416 ms 2022-03-05 09:44:23,891 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df. because 80a1ffe1dc2d8f258a6367d204dc64df/e has an old edit so flush to free WALs after random delay 191478 ms 2022-03-05 09:44:33,892 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. because 7d1b53259f1d8280a47f7704f98f2c9f/e has an old edit so flush to free WALs after random delay 148529 ms 2022-03-05 09:44:36,780 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 9b121c22e18472616debd0996ee2a4cb 1/1 column families, dataSize=553.14 KB heapSize=1020.11 KB 2022-03-05 09:44:36,844 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=553.14 KB at sequenceid=10965 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/.tmp/e/c5d6aebf6dbe4479b64d1ace32047306 2022-03-05 09:44:36,871 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/c5d6aebf6dbe4479b64d1ace32047306, entries=6375, sequenceid=10965, filesize=66.8 K 2022-03-05 09:44:36,873 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~553.14 KB/566413, heapSize ~1020.09 KB/1044576, currentSize=0 B/0 for 9b121c22e18472616debd0996ee2a4cb in 93ms, sequenceid=10965, compaction requested=true 2022-03-05 09:44:36,874 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 9b121c22e18472616debd0996ee2a4cb/e in aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb. 2022-03-05 09:44:36,874 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/8f6a51bacef345debad6a0e6ceb856e5, hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/25b4aad0d2bb4e18b44232186316eaa5, hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/c5d6aebf6dbe4479b64d1ace32047306] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/.tmp, totalSize=497.6 K 2022-03-05 09:44:37,023 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 9b121c22e18472616debd0996ee2a4cb#e#compaction#55 average throughput is 48.27 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 09:44:37,077 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 9b121c22e18472616debd0996ee2a4cb/e of 9b121c22e18472616debd0996ee2a4cb into 4f32e90c300b4130a465533983e52667(size=508.5 K), total size for store is 508.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 09:44:37,077 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., storeName=9b121c22e18472616debd0996ee2a4cb/e, priority=13, startTime=1646473476874; duration=0sec 2022-03-05 09:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=434.10 KB, freeSize=306.78 MB, max=307.20 MB, blockCount=22, accesses=6238, hits=5772, hitRatio=92.53%, , cachingAccesses=5804, cachingHits=5766, cachingHitsRatio=99.35%, evictions=2222, evicted=0, evictedPerRun=0.0 2022-03-05 09:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 09:47:02,422 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 7d1b53259f1d8280a47f7704f98f2c9f 1/1 column families, dataSize=671.56 KB heapSize=1.13 MB 2022-03-05 09:47:02,494 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=671.56 KB at sequenceid=7771 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/.tmp/e/0ed6c9a89a5c4002805e4c1fa597ab2b 2022-03-05 09:47:02,508 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/0ed6c9a89a5c4002805e4c1fa597ab2b, entries=6508, sequenceid=7771, filesize=64.4 K 2022-03-05 09:47:02,509 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~671.56 KB/687677, heapSize ~1.13 MB/1182056, currentSize=0 B/0 for 7d1b53259f1d8280a47f7704f98f2c9f in 88ms, sequenceid=7771, compaction requested=true 2022-03-05 09:47:02,510 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 7d1b53259f1d8280a47f7704f98f2c9f/e in aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. 2022-03-05 09:47:02,510 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/12e47a2953e243249e8100a6f89b5472, hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/78d1b77deeb747c0a9bd912c715f15d3, hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/0ed6c9a89a5c4002805e4c1fa597ab2b] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/.tmp, totalSize=495.1 K 2022-03-05 09:47:02,609 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 7d1b53259f1d8280a47f7704f98f2c9f#e#compaction#57 average throughput is 53.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 09:47:02,656 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 7d1b53259f1d8280a47f7704f98f2c9f/e of 7d1b53259f1d8280a47f7704f98f2c9f into 0be95bb2a69b403aa8bcf36844017561(size=549.0 K), total size for store is 549.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 09:47:02,656 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., storeName=7d1b53259f1d8280a47f7704f98f2c9f/e, priority=13, startTime=1646473622510; duration=0sec 2022-03-05 09:47:35,369 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 80a1ffe1dc2d8f258a6367d204dc64df 1/1 column families, dataSize=306.70 KB heapSize=573.24 KB 2022-03-05 09:47:36,496 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=306.70 KB at sequenceid=8082 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/.tmp/e/d41f9d846cf24e36988270c938ed4fa8 2022-03-05 09:47:36,713 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/d41f9d846cf24e36988270c938ed4fa8, entries=3637, sequenceid=8082, filesize=41.0 K 2022-03-05 09:47:36,735 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~306.70 KB/314059, heapSize ~573.23 KB/586984, currentSize=0 B/0 for 80a1ffe1dc2d8f258a6367d204dc64df in 1366ms, sequenceid=8082, compaction requested=true 2022-03-05 09:47:36,736 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 80a1ffe1dc2d8f258a6367d204dc64df/e in aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df. 2022-03-05 09:47:36,736 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/2acc4b57dd154eff9d74e3f65b375b2b, hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/d96edbddb0c44e53a3ecb754c1677800, hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/d41f9d846cf24e36988270c938ed4fa8] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/.tmp, totalSize=301.1 K 2022-03-05 09:47:37,291 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 80a1ffe1dc2d8f258a6367d204dc64df#e#compaction#59 average throughput is 21.00 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 09:47:37,417 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 80a1ffe1dc2d8f258a6367d204dc64df/e of 80a1ffe1dc2d8f258a6367d204dc64df into d6a72e9bdc87474492d51122310397fb(size=298.7 K), total size for store is 298.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 09:47:37,417 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., storeName=80a1ffe1dc2d8f258a6367d204dc64df/e, priority=13, startTime=1646473656736; duration=0sec 2022-03-05 09:47:37,792 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646473054195.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646473657776.meta 2022-03-05 09:47:37,792 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646473054195.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646473054195.meta 2022-03-05 09:47:43,903 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e. because 4333b489291b1f77671b8f2e78779a1e/__status_family___ has an old edit so flush to free WALs after random delay 227072 ms 2022-03-05 09:47:48,859 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 7534b3b1812808d2d3c16342fbf347a5 1/1 column families, dataSize=188.69 KB heapSize=351.71 KB 2022-03-05 09:47:48,909 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=188.69 KB at sequenceid=6494 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/.tmp/e/f6d45b55baa640c6bce1a9192419bb27 2022-03-05 09:47:48,922 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/f6d45b55baa640c6bce1a9192419bb27, entries=2196, sequenceid=6494, filesize=26.2 K 2022-03-05 09:47:48,923 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~188.69 KB/193216, heapSize ~351.70 KB/360136, currentSize=0 B/0 for 7534b3b1812808d2d3c16342fbf347a5 in 64ms, sequenceid=6494, compaction requested=false 2022-03-05 09:48:03,508 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 756 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 09:48:03,508 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 757 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 09:48:03,508 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 756 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 09:48:03,508 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 596 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 09:48:45,681 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 873 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 09:48:51,009 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6103 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 09:48:51,009 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4705 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 09:48:59,762 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 11311 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 09:48:59,762 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Requesting log roll because we exceeded slow sync threshold; time=11311 ms, threshold=10000 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 09:48:59,762 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646473726732","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":13030,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 09:48:59,763 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646473724905","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":14206,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 09:48:59,855 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646473728450","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":11405,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":339,"class":"HRegionServer","multi.mutations":1} 2022-03-05 09:48:59,903 WARN [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 12324ms No GCs detected 2022-03-05 09:48:59,989 INFO [regionserver/cvp328:16201.Chore.3] hbase.ScheduledChore: Chore: CompactionChecker missed its start time 2022-03-05 09:48:59,989 INFO [regionserver/cvp328:16201.Chore.3] hbase.ScheduledChore: Chore: MemstoreFlusherChore missed its start time 2022-03-05 09:49:00,141 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646473726303","responsesize":"90","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 13 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x06Kernel ","processingtimems":13837,"client":"172.31.0.219:43118","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":13} 2022-03-05 09:49:00,912 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646473286132 with entries=1007, filesize=455 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646473739902 2022-03-05 09:49:09,989 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb. because ccdacf4b414cb8d1bc548efc77c663cb/e has an old edit so flush to free WALs after random delay 221876 ms 2022-03-05 09:49:10,966 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646473657776.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646473750913.meta 2022-03-05 09:49:10,966 WARN [Close-WAL-Writer-52] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 09:49:10,980 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646473657776.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646473657776.meta 2022-03-05 09:49:10,980 WARN [Close-WAL-Writer-52] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 09:49:10,981 INFO [Close-WAL-Writer-52] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646473657776.meta 2022-03-05 09:49:11,019 INFO [Close-WAL-Writer-52] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646473657776.meta after 38ms 2022-03-05 09:49:15,021 WARN [Close-WAL-Writer-52] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646473657776.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646473657776.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 09:51:30,976 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 4333b489291b1f77671b8f2e78779a1e 1/1 column families, dataSize=7.99 KB heapSize=12.75 KB 2022-03-05 09:51:31,020 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=7.99 KB at sequenceid=461 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/.tmp/__status_family___/69725697445d4cac8df314f18a8d0f41 2022-03-05 09:51:31,030 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/69725697445d4cac8df314f18a8d0f41, entries=3, sequenceid=461, filesize=5.1 K 2022-03-05 09:51:31,031 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~7.99 KB/8184, heapSize ~12.73 KB/13040, currentSize=0 B/0 for 4333b489291b1f77671b8f2e78779a1e in 55ms, sequenceid=461, compaction requested=false 2022-03-05 09:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=513.71 KB, freeSize=306.70 MB, max=307.20 MB, blockCount=23, accesses=6399, hits=5824, hitRatio=91.01%, , cachingAccesses=5861, cachingHits=5816, cachingHitsRatio=99.23%, evictions=2252, evicted=0, evictedPerRun=0.0 2022-03-05 09:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 09:52:51,865 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing ccdacf4b414cb8d1bc548efc77c663cb 1/1 column families, dataSize=305.32 KB heapSize=530.20 KB 2022-03-05 09:52:51,949 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=305.32 KB at sequenceid=4447 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/.tmp/e/8063275f6b814c4ea9c7b7e424bec44f 2022-03-05 09:52:51,963 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/8063275f6b814c4ea9c7b7e424bec44f, entries=3051, sequenceid=4447, filesize=31.6 K 2022-03-05 09:52:51,964 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~305.32 KB/312644, heapSize ~530.19 KB/542912, currentSize=0 B/0 for ccdacf4b414cb8d1bc548efc77c663cb in 99ms, sequenceid=4447, compaction requested=true 2022-03-05 09:52:51,964 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of ccdacf4b414cb8d1bc548efc77c663cb/e in aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb. 2022-03-05 09:52:51,964 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/5f6d49dcd992478ca0cb9e16621ac5d7, hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/2ac85f3be0b243d9b9a0eb3277c1460d, hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/8063275f6b814c4ea9c7b7e424bec44f] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/.tmp, totalSize=265.0 K 2022-03-05 09:52:52,015 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: ccdacf4b414cb8d1bc548efc77c663cb#e#compaction#63 average throughput is 57.01 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 09:52:52,064 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in ccdacf4b414cb8d1bc548efc77c663cb/e of ccdacf4b414cb8d1bc548efc77c663cb into 4927e58e7534492a9d34d845b5af0c95(size=277.5 K), total size for store is 277.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 09:52:52,064 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., storeName=ccdacf4b414cb8d1bc548efc77c663cb/e, priority=13, startTime=1646473971964; duration=0sec 2022-03-05 09:53:40,285 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391. because 0a764e51258807790269e46c55b99391/e has an old edit so flush to free WALs after random delay 207490 ms 2022-03-05 09:56:05,386 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4828 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 09:56:05,436 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4104ms No GCs detected 2022-03-05 09:56:05,451 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4616 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 09:56:11,796 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1380 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 09:56:12,096 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 732 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 09:56:18,352 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 537 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 09:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=534.34 KB, freeSize=306.68 MB, max=307.20 MB, blockCount=23, accesses=6483, hits=5872, hitRatio=90.58%, , cachingAccesses=5911, cachingHits=5864, cachingHitsRatio=99.20%, evictions=2282, evicted=0, evictedPerRun=0.0 2022-03-05 09:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 09:57:07,775 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 0a764e51258807790269e46c55b99391 1/1 column families, dataSize=950.60 KB heapSize=1.67 MB 2022-03-05 09:57:07,848 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=950.60 KB at sequenceid=14856 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/.tmp/e/95150a6d13054c09a990b2df5640aac6 2022-03-05 09:57:07,864 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/95150a6d13054c09a990b2df5640aac6, entries=10237, sequenceid=14856, filesize=93.4 K 2022-03-05 09:57:07,865 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~950.60 KB/973417, heapSize ~1.67 MB/1749912, currentSize=0 B/0 for 0a764e51258807790269e46c55b99391 in 90ms, sequenceid=14856, compaction requested=true 2022-03-05 09:57:07,866 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 0a764e51258807790269e46c55b99391/e in aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391. 2022-03-05 09:57:07,866 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/e0866d8a50c74b098dfedf756fb9610e, hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/329674adf6d54e4a9f69ea5368d06bf1, hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/95150a6d13054c09a990b2df5640aac6] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/.tmp, totalSize=697.7 K 2022-03-05 09:57:07,983 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 0a764e51258807790269e46c55b99391#e#compaction#65 average throughput is 57.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 09:57:08,034 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 0a764e51258807790269e46c55b99391/e of 0a764e51258807790269e46c55b99391 into a822dd0efb984cbf96d685d69058f1a1(size=765.9 K), total size for store is 765.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 09:57:08,034 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391., storeName=0a764e51258807790269e46c55b99391/e, priority=13, startTime=1646474227865; duration=0sec 2022-03-05 09:59:05,477 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646473739902 with entries=1332, filesize=577.42 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646474345455 2022-03-05 09:59:05,477 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646469534720 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646469534720 2022-03-05 09:59:05,483 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646470135323 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646470135323 2022-03-05 09:59:05,489 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646470735728 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646470735728 2022-03-05 09:59:05,496 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646471336564 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646471336564 2022-03-05 09:59:05,501 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646471936613 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646471936613 2022-03-05 09:59:05,509 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646472362610 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646472362610 2022-03-05 09:59:05,515 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646472967986 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646472967986 2022-03-05 09:59:05,536 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646473043904 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646473043904 2022-03-05 09:59:15,525 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2658 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 09:59:15,525 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 749 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 09:59:15,525 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 624 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 09:59:17,025 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1022 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 09:59:17,025 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 849 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 09:59:17,127 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646473750913.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646474355477.meta 2022-03-05 09:59:17,127 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646473750913.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646473750913.meta 2022-03-05 09:59:17,179 WARN [Close-WAL-Writer-53] asyncfs.FanOutOneBlockAsyncDFSOutputHelper: complete file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646473750913.meta not finished, retry = 0 2022-03-05 10:00:05,345 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1349ms No GCs detected 2022-03-05 10:00:07,509 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1664ms No GCs detected 2022-03-05 10:01:08,435 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2342 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:01:08,435 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 778 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:01:12,296 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3242 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:01:12,311 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1790 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:01:12,311 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 991 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:01:24,018 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 795 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:01:32,493 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1958ms No GCs detected 2022-03-05 10:01:35,416 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:01:35,534 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:01:35,534 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:01:35,534 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:01:35,534 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:01:35,534 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:01:35,534 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:01:35,534 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:01:35,534 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:01:35,535 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:01:35,548 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:01:35,548 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:01:35,614 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: rg.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 10:01:38,243 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646474345455 with entries=306, filesize=137.85 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646474495542 2022-03-05 10:01:38,243 WARN [Close-WAL-Writer-53] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:01:38,243 WARN [Close-WAL-Writer-53] wal.AsyncProtobufLogWriter: normal close failed, try recover java.lang.IllegalStateException: should call flush first before calling close at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:510) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:514) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:01:38,243 INFO [Close-WAL-Writer-53] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646474345455 2022-03-05 10:01:38,260 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646474486779","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":11481,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":90,"class":"HRegionServer","multi.mutations":1} 2022-03-05 10:01:38,260 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646474487978","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":10282,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 10:01:38,261 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646474488065","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":10196,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 10:01:38,277 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646474485993","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":12267,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 10:01:38,268 INFO [Close-WAL-Writer-53] retry.RetryInvocationHandler: .apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.recoverLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 10:01:38,301 INFO [Close-WAL-Writer-53] retry.RetryInvocationHandler: menodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.recoverLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 1 failover attempts. Trying to failover after sleeping for 355ms. 2022-03-05 10:01:38,659 INFO [Close-WAL-Writer-53] retry.RetryInvocationHandler: menodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.recoverLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 2 failover attempts. Trying to failover after sleeping for 945ms. 2022-03-05 10:01:39,621 INFO [Close-WAL-Writer-53] retry.RetryInvocationHandler: enodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.recoverLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 3 failover attempts. Trying to failover after sleeping for 1413ms. 2022-03-05 10:01:39,887 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 10:01:41,036 WARN [Close-WAL-Writer-53] retry.RetryInvocationHandler: A failover has occurred since the start of call #3987 ClientNamenodeProtocolTranslatorPB.recoverLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 2022-03-05 10:01:41,173 INFO [Close-WAL-Writer-53] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646474345455 after 2930ms 2022-03-05 10:01:44,183 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 508 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:01:44,528 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=496.80 KB, freeSize=306.71 MB, max=307.20 MB, blockCount=20, accesses=6624, hits=5915, hitRatio=89.30%, , cachingAccesses=5953, cachingHits=5904, cachingHitsRatio=99.18%, evictions=2312, evicted=0, evictedPerRun=0.0 2022-03-05 10:01:45,174 INFO [Close-WAL-Writer-53] util.RecoverLeaseFSUtils: Recovered lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646474345455 after 6931ms 2022-03-05 10:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 10:01:48,279 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646474355477.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646474508243.meta 2022-03-05 10:01:48,279 WARN [Close-WAL-Writer-54] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:01:48,279 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646474355477.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646474355477.meta 2022-03-05 10:01:48,279 WARN [Close-WAL-Writer-54] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:01:48,280 INFO [Close-WAL-Writer-54] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646474355477.meta 2022-03-05 10:01:48,307 INFO [Close-WAL-Writer-54] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646474355477.meta after 27ms 2022-03-05 10:01:52,309 WARN [Close-WAL-Writer-54] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646474355477.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646474355477.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 10:02:16,278 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1025ms No GCs detected 2022-03-05 10:04:16,108 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 9388ms No GCs detected 2022-03-05 10:04:16,115 INFO [regionserver/cvp328:16201.Chore.1] hbase.ScheduledChore: Chore: MemstoreFlusherChore missed its start time 2022-03-05 10:04:16,115 INFO [regionserver/cvp328:16201.Chore.1] hbase.ScheduledChore: Chore: CompactionChecker missed its start time 2022-03-05 10:04:16,180 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:04:16,547 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646474495542 with entries=364, filesize=155.56 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646474656245 2022-03-05 10:04:16,548 WARN [Close-WAL-Writer-54] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:04:16,548 WARN [Close-WAL-Writer-54] wal.AsyncProtobufLogWriter: normal close failed, try recover java.lang.IllegalStateException: should call flush first before calling close at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:510) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:514) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:04:16,548 INFO [Close-WAL-Writer-54] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646474495542 2022-03-05 10:04:16,553 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646474645659","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":10894,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 10:04:16,579 INFO [Close-WAL-Writer-54] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646474495542 after 31ms 2022-03-05 10:04:17,151 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 597 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:04:20,580 INFO [Close-WAL-Writer-54] util.RecoverLeaseFSUtils: Recovered lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646474495542 after 4032ms 2022-03-05 10:04:25,568 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1903ms No GCs detected 2022-03-05 10:04:26,148 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3558 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:04:26,148 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3153 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:04:26,573 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646474508243.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646474666548.meta 2022-03-05 10:04:26,573 WARN [Close-WAL-Writer-55] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:04:26,574 WARN [Close-WAL-Writer-55] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:04:26,574 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646474508243.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646474508243.meta 2022-03-05 10:04:26,574 INFO [Close-WAL-Writer-55] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646474508243.meta 2022-03-05 10:04:26,581 INFO [Close-WAL-Writer-55] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646474508243.meta after 7ms 2022-03-05 10:04:30,583 WARN [Close-WAL-Writer-55] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646474508243.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646474508243.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 10:04:34,680 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2486ms No GCs detected 2022-03-05 10:04:34,992 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 958 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:05:12,712 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3784ms No GCs detected 2022-03-05 10:05:12,783 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3038 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:05:16,739 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 10:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=511.11 KB, freeSize=306.70 MB, max=307.20 MB, blockCount=21, accesses=6652, hits=5942, hitRatio=89.33%, , cachingAccesses=5981, cachingHits=5931, cachingHitsRatio=99.16%, evictions=2341, evicted=0, evictedPerRun=0.0 2022-03-05 10:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 10:09:12,040 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1207 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:09:54,572 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2451 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:09:54,638 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1647 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:10:18,661 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2827ms No GCs detected 2022-03-05 10:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=511.11 KB, freeSize=306.70 MB, max=307.20 MB, blockCount=21, accesses=6674, hits=5964, hitRatio=89.36%, , cachingAccesses=6003, cachingHits=5953, cachingHitsRatio=99.17%, evictions=2371, evicted=0, evictedPerRun=0.0 2022-03-05 10:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 10:12:01,683 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 605 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:12:11,567 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2471 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:12:11,567 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1161 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:12:33,989 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 970 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:12:36,534 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2153 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:12:36,535 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1337ms No GCs detected 2022-03-05 10:12:46,511 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2962ms No GCs detected 2022-03-05 10:12:48,219 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1157ms No GCs detected 2022-03-05 10:12:49,218 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 554 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:12:50,300 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 733 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:13:00,051 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1593 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:13:00,051 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1593 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:13:21,960 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1648ms No GCs detected 2022-03-05 10:13:40,969 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1208 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:13:40,969 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1868 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:13:40,969 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1855 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:13:43,341 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1025 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:14:02,373 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 795 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:14:18,236 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1338ms No GCs detected 2022-03-05 10:14:19,262 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2859 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:14:19,262 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2858 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:14:21,792 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:568) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.addBlock over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 10:14:21,935 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646474656245 with entries=1299, filesize=577.40 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646475259275 2022-03-05 10:14:32,034 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646474666548.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646475271936.meta 2022-03-05 10:14:32,034 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646474666548.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646474666548.meta 2022-03-05 10:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=511.11 KB, freeSize=306.70 MB, max=307.20 MB, blockCount=21, accesses=6696, hits=5986, hitRatio=89.40%, , cachingAccesses=6025, cachingHits=5975, cachingHitsRatio=99.17%, evictions=2401, evicted=0, evictedPerRun=0.0 2022-03-05 10:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 10:16:49,022 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1449 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:16:49,022 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 962 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=511.11 KB, freeSize=306.70 MB, max=307.20 MB, blockCount=21, accesses=6718, hits=6008, hitRatio=89.43%, , cachingAccesses=6047, cachingHits=5997, cachingHitsRatio=99.17%, evictions=2431, evicted=0, evictedPerRun=0.0 2022-03-05 10:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 10:21:53,573 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1453ms No GCs detected 2022-03-05 10:21:59,817 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.sync(AsyncProtobufLogWriter.java:148) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:394) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.consume(AsyncFSWAL.java:581) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:22:00,107 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646475259275 with entries=964, filesize=431.70 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646475719936 2022-03-05 10:22:00,107 WARN [Close-WAL-Writer-56] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:22:00,108 WARN [Close-WAL-Writer-56] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:22:00,108 INFO [Close-WAL-Writer-56] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646475259275 2022-03-05 10:22:00,117 INFO [Close-WAL-Writer-56] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646475259275 after 9ms 2022-03-05 10:22:04,118 INFO [Close-WAL-Writer-56] util.RecoverLeaseFSUtils: Recovered lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646475259275 after 4010ms 2022-03-05 10:22:07,292 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1052ms No GCs detected 2022-03-05 10:22:11,222 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646475271936.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646475730107.meta 2022-03-05 10:22:11,222 WARN [Close-WAL-Writer-57] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:22:11,222 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646475271936.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646475271936.meta 2022-03-05 10:22:11,222 WARN [Close-WAL-Writer-57] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:22:11,222 INFO [Close-WAL-Writer-57] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646475271936.meta 2022-03-05 10:22:11,584 INFO [Close-WAL-Writer-57] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646475271936.meta after 362ms 2022-03-05 10:22:15,586 WARN [Close-WAL-Writer-57] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646475271936.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646475271936.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 10:22:40,843 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1295ms No GCs detected 2022-03-05 10:22:42,714 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4406 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:22:42,714 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3525 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:22:43,462 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 509 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:22:56,088 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1311 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:22:57,625 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1537 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:22:58,729 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 580 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:22:58,729 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 579 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:23:00,575 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 773 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:23:00,575 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 767 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:23:04,030 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1143ms No GCs detected 2022-03-05 10:23:13,957 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4892 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:23:18,528 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 7172ms No GCs detected 2022-03-05 10:23:18,617 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 9298 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:23:18,617 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 8024 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:23:20,503 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1745 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:23:21,625 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1525 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:23:21,679 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1175 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:23:41,090 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 694 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:23:45,303 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2110ms No GCs detected 2022-03-05 10:23:45,988 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 674 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:23:47,065 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1087 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:23:49,300 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1197ms No GCs detected 2022-03-05 10:23:50,893 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1092ms No GCs detected 2022-03-05 10:23:50,968 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3365 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:23:50,968 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2782 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:23:50,968 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2781 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:24:12,483 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 603 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:24:12,483 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 574 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:25:59,208 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2189 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:25:59,208 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2164 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:25:59,208 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2088 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:25:59,209 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1865 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:26:01,814 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 977 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:26:19,390 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1032ms No GCs detected 2022-03-05 10:26:21,948 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1557ms No GCs detected 2022-03-05 10:26:24,694 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1329ms No GCs detected 2022-03-05 10:26:24,727 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 9206 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:26:24,928 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 8820 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:26:24,928 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 7550 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:26:24,928 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 7131 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:26:24,928 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1808 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:26:28,262 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 617 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=511.11 KB, freeSize=306.70 MB, max=307.20 MB, blockCount=21, accesses=6740, hits=6030, hitRatio=89.47%, , cachingAccesses=6069, cachingHits=6019, cachingHitsRatio=99.18%, evictions=2461, evicted=0, evictedPerRun=0.0 2022-03-05 10:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 10:27:15,825 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3684ms No GCs detected 2022-03-05 10:27:15,826 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4184 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:27:15,826 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3730 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:27:18,322 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1421 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:27:32,994 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 10:28:12,824 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 775 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:29:05,017 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1826ms No GCs detected 2022-03-05 10:30:41,554 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1177 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=511.11 KB, freeSize=306.70 MB, max=307.20 MB, blockCount=21, accesses=6762, hits=6052, hitRatio=89.50%, , cachingAccesses=6091, cachingHits=6041, cachingHitsRatio=99.18%, evictions=2490, evicted=0, evictedPerRun=0.0 2022-03-05 10:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-05 10:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 10:32:05,714 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646475719936 with entries=1329, filesize=587.73 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646476325696 2022-03-05 10:32:14,285 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1995ms No GCs detected 2022-03-05 10:32:16,464 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646475730107.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646476335715.meta 2022-03-05 10:32:16,469 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646475730107.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646475730107.meta 2022-03-05 10:36:44,825 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1872ms No GCs detected 2022-03-05 10:36:53,110 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 7784ms No GCs detected 2022-03-05 10:36:53,127 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=511.11 KB, freeSize=306.70 MB, max=307.20 MB, blockCount=21, accesses=6790, hits=6080, hitRatio=89.54%, , cachingAccesses=6119, cachingHits=6069, cachingHitsRatio=99.18%, evictions=2521, evicted=0, evictedPerRun=0.0 2022-03-05 10:36:53,130 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 10:36:53,238 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 10785 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:36:53,239 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Requesting log roll because we exceeded slow sync threshold; time=10785 ms, threshold=10000 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:36:53,239 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Connection to 172.31.0.219/172.31.0.219:15010 closed at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$channelInactive$2(FanOutOneBlockAsyncDFSOutput.java:286) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.channelInactive(FanOutOneBlockAsyncDFSOutput.java:285) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:389) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:354) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelInactive(IdleStateHandler.java:277) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:831) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:36:53,239 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646476602453","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":10786,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-05 10:36:53,556 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646476325696 with entries=584, filesize=256.68 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646476613242 2022-03-05 10:36:53,556 WARN [Close-WAL-Writer-58] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:36:53,556 WARN [Close-WAL-Writer-58] wal.AsyncProtobufLogWriter: normal close failed, try recover java.lang.IllegalStateException: should call flush first before calling close at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:510) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:514) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:36:53,557 INFO [Close-WAL-Writer-58] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646476325696 2022-03-05 10:36:53,605 INFO [Close-WAL-Writer-58] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646476325696 after 48ms 2022-03-05 10:36:57,611 INFO [Close-WAL-Writer-58] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646476325696 after 4054ms 2022-03-05 10:37:09,187 INFO [Close-WAL-Writer-58] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=2 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646476325696 after 15630ms 2022-03-05 10:37:23,576 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646476335715.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646476643557.meta 2022-03-05 10:37:23,576 WARN [Close-WAL-Writer-59] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:37:23,577 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646476335715.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646476335715.meta 2022-03-05 10:37:23,577 WARN [Close-WAL-Writer-59] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:37:23,577 INFO [Close-WAL-Writer-59] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646476335715.meta 2022-03-05 10:37:23,583 INFO [Close-WAL-Writer-59] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646476335715.meta after 6ms 2022-03-05 10:37:27,587 WARN [Close-WAL-Writer-59] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646476335715.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646476335715.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 10:37:31,214 INFO [Close-WAL-Writer-58] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=3 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646476325696 after 37657ms 2022-03-05 10:37:31,215 WARN [Close-WAL-Writer-58] util.RecoverLeaseFSUtils: Cannot recoverLease after trying for 23000ms (hbase.lease.recovery.timeout); continuing, but may be DATALOSS!!!; attempt=3 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646476325696 after 37658ms 2022-03-05 10:37:55,344 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 553 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:37:59,235 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 988 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:41:17,087 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 978 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:41:34,290 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1379ms No GCs detected 2022-03-05 10:41:34,314 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3302 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=511.11 KB, freeSize=306.70 MB, max=307.20 MB, blockCount=21, accesses=6812, hits=6102, hitRatio=89.58%, , cachingAccesses=6141, cachingHits=6091, cachingHitsRatio=99.19%, evictions=2550, evicted=0, evictedPerRun=0.0 2022-03-05 10:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 10:43:35,519 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 523 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:43:44,304 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 6277ms No GCs detected 2022-03-05 10:43:44,476 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 10:43:46,914 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1919ms No GCs detected 2022-03-05 10:43:46,916 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2374 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:43:46,916 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2005 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:43:46,916 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1998 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:43:46,918 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 341ms. 2022-03-05 10:43:47,668 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 751 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:43:55,784 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 533 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:43:57,919 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1844 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:43:57,919 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1451 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:44:00,109 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1997 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:44:04,841 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1877ms No GCs detected 2022-03-05 10:44:07,844 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5379 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:44:07,845 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2488 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:44:07,845 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2487 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:44:07,845 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1409 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:44:16,070 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2838 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:44:17,634 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 10:44:25,171 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646476643557.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477065138.meta 2022-03-05 10:44:25,171 WARN [Close-WAL-Writer-60] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:44:25,171 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646476643557.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646476643557.meta 2022-03-05 10:44:25,171 WARN [Close-WAL-Writer-60] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:44:25,171 INFO [Close-WAL-Writer-60] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646476643557.meta 2022-03-05 10:44:25,181 INFO [Close-WAL-Writer-60] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646476643557.meta after 10ms 2022-03-05 10:44:29,182 WARN [Close-WAL-Writer-60] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646476643557.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646476643557.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 10:44:46,116 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb. because 9b121c22e18472616debd0996ee2a4cb/e has an old edit so flush to free WALs after random delay 52203 ms 2022-03-05 10:44:59,307 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1352 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:45:00,732 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2628 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:45:00,732 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2235 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:45:03,838 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1634ms No GCs detected 2022-03-05 10:45:06,448 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.MetricsWAL: AsyncFSWAL-0-hdfs://mycluster/hbase took 1635 ms appending an edit to wal; len~=193 2022-03-05 10:45:14,897 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2937ms No GCs detected 2022-03-05 10:45:24,233 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 5238ms No GCs detected 2022-03-05 10:45:24,360 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.sync(AsyncProtobufLogWriter.java:148) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:394) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.consume(AsyncFSWAL.java:581) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:45:26,835 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646476613242 with entries=1117, filesize=494.79 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477124574 2022-03-05 10:45:26,836 WARN [Close-WAL-Writer-59] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:45:26,836 WARN [Close-WAL-Writer-59] wal.AsyncProtobufLogWriter: normal close failed, try recover java.lang.IllegalStateException: should call flush first before calling close at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:510) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:514) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:45:26,836 INFO [Close-WAL-Writer-59] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646476613242 2022-03-05 10:45:26,837 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646477110363","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":16474,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 10:45:26,837 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646477111309","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":15528,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 10:45:26,837 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646477111733","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":15104,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 10:45:26,838 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646477116276","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":10562,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-05 10:45:26,838 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646477111326","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x99\\xC4\\x07Devi ","processingtimems":15512,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 10:45:26,838 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646477116276","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":10562,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 10:45:26,852 INFO [Close-WAL-Writer-59] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646476613242 after 16ms 2022-03-05 10:45:26,912 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477065138.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477126836.meta 2022-03-05 10:45:26,912 WARN [Close-WAL-Writer-60] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:45:26,912 WARN [Close-WAL-Writer-60] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:45:26,913 INFO [Close-WAL-Writer-60] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477065138.meta 2022-03-05 10:45:26,913 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477065138.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477065138.meta 2022-03-05 10:45:26,952 INFO [Log-Archiver-0] retry.RetryInvocationHandler: at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setTimes(ClientNamenodeProtocolServerSideTranslatorPB.java:1062) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.setTimes over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 10:45:26,952 INFO [Close-WAL-Writer-60] retry.RetryInvocationHandler: .apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.recoverLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 10:45:26,952 WARN [Close-WAL-Writer-60] retry.RetryInvocationHandler: A failover has occurred since the start of call #4191 ClientNamenodeProtocolTranslatorPB.recoverLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 2022-03-05 10:45:26,955 INFO [Log-Archiver-0] retry.RetryInvocationHandler: lientNamenodeProtocolServerSideTranslatorPB.setTimes(ClientNamenodeProtocolServerSideTranslatorPB.java:1062) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.setTimes over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 1 failover attempts. Trying to failover after sleeping for 610ms. 2022-03-05 10:45:26,955 INFO [Close-WAL-Writer-60] retry.RetryInvocationHandler: menodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.recoverLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 1 failover attempts. Trying to failover after sleeping for 801ms. 2022-03-05 10:45:27,566 INFO [Log-Archiver-0] retry.RetryInvocationHandler: ientNamenodeProtocolServerSideTranslatorPB.setTimes(ClientNamenodeProtocolServerSideTranslatorPB.java:1062) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.setTimes over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 2 failover attempts. Trying to failover after sleeping for 1081ms. 2022-03-05 10:45:27,757 WARN [Close-WAL-Writer-60] retry.RetryInvocationHandler: A failover has occurred since the start of call #4191 ClientNamenodeProtocolTranslatorPB.recoverLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 2022-03-05 10:45:27,758 INFO [Close-WAL-Writer-60] retry.RetryInvocationHandler: enodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.recoverLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 2 failover attempts. Trying to failover after sleeping for 1570ms. 2022-03-05 10:45:29,328 WARN [Close-WAL-Writer-60] retry.RetryInvocationHandler: A failover has occurred since the start of call #4191 ClientNamenodeProtocolTranslatorPB.recoverLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 2022-03-05 10:45:29,331 WARN [Close-WAL-Writer-60] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477065138.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477065138.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 10:45:30,861 INFO [Close-WAL-Writer-59] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646476613242 after 4025ms 2022-03-05 10:45:38,319 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 9b121c22e18472616debd0996ee2a4cb 1/1 column families, dataSize=560.41 KB heapSize=1.01 MB 2022-03-05 10:45:38,369 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=560.41 KB at sequenceid=12647 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/.tmp/e/66df821cd4fe4e3aa1f21097c8efd58b 2022-03-05 10:45:38,383 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/66df821cd4fe4e3aa1f21097c8efd58b, entries=6475, sequenceid=12647, filesize=68.5 K 2022-03-05 10:45:38,384 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~560.41 KB/573855, heapSize ~1.01 MB/1059656, currentSize=0 B/0 for 9b121c22e18472616debd0996ee2a4cb in 66ms, sequenceid=12647, compaction requested=false 2022-03-05 10:45:52,341 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2448 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:45:52,718 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 592 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:45:53,673 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 686 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:45:57,556 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2772 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:46:04,454 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2065ms No GCs detected 2022-03-05 10:46:04,457 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2121 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:46:11,314 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 8426 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:46:22,880 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 658 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:46:26,454 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1640 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:46:26,454 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1598 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:46:27,160 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 692 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:46:29,392 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 10:46:31,647 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp328.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp328.sjc.aristanetworks.com/172.30.41.118:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 707ms. 2022-03-05 10:46:33,211 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3317ms No GCs detected 2022-03-05 10:46:36,261 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: tNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 2 failover attempts. Trying to failover after sleeping for 1489ms. 2022-03-05 10:46:36,285 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 8615 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:46:36,285 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 8392 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:46:36,285 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 8206 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:46:36,285 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 7825 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:46:36,285 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 7825 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:46:36,285 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3101 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:46:36,285 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3078 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:46:36,285 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3077 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:46:36,285 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2797 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:46:36,285 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2796 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:46:36,285 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2790 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:46:36,285 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2103 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:46:36,294 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1680 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:46:51,261 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=511.11 KB, freeSize=306.70 MB, max=307.20 MB, blockCount=21, accesses=6831, hits=6121, hitRatio=89.61%, , cachingAccesses=6160, cachingHits=6110, cachingHitsRatio=99.19%, evictions=2580, evicted=0, evictedPerRun=0.0 2022-03-05 10:46:51,319 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 6943ms No GCs detected 2022-03-05 10:46:51,372 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:46:51,318 INFO [regionserver/cvp328:16201.Chore.3] hbase.ScheduledChore: Chore: CompactionChecker missed its start time 2022-03-05 10:46:51,267 INFO [regionserver/cvp328:16201.Chore.1] hbase.ScheduledChore: Chore: MemstoreFlusherChore missed its start time 2022-03-05 10:46:51,427 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 10:46:51,830 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: rg.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 10:46:55,667 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3820ms No GCs detected 2022-03-05 10:46:55,668 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 1 failover attempts. Trying to failover after sleeping for 689ms. 2022-03-05 10:46:56,360 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: amenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 2 failover attempts. Trying to failover after sleeping for 1529ms. 2022-03-05 10:46:58,077 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477124574 with entries=196, filesize=91.87 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477211445 2022-03-05 10:46:58,077 WARN [Close-WAL-Writer-60] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:46:58,078 WARN [Close-WAL-Writer-60] wal.AsyncProtobufLogWriter: normal close failed, try recover java.lang.IllegalStateException: should call flush first before calling close at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:510) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:514) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:46:58,078 INFO [Close-WAL-Writer-60] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477124574 2022-03-05 10:46:58,092 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646477201782","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":16310,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 10:46:58,118 INFO [Close-WAL-Writer-60] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477124574 after 40ms 2022-03-05 10:47:00,286 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 867 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:47:00,286 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 746 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:47:01,245 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 856 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:47:01,245 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 738 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:47:02,127 INFO [Close-WAL-Writer-60] util.RecoverLeaseFSUtils: Recovered lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477124574 after 4049ms 2022-03-05 10:47:06,255 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2077 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:47:06,256 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2042 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:47:06,256 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 622 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:47:08,143 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477126836.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477228077.meta 2022-03-05 10:47:08,143 WARN [Close-WAL-Writer-61] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:47:08,143 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477126836.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477126836.meta 2022-03-05 10:47:08,144 WARN [Close-WAL-Writer-61] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:47:08,144 INFO [Close-WAL-Writer-61] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477126836.meta 2022-03-05 10:47:08,292 INFO [Close-WAL-Writer-61] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477126836.meta after 148ms 2022-03-05 10:47:11,964 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. because 7d1b53259f1d8280a47f7704f98f2c9f/e has an old edit so flush to free WALs after random delay 198767 ms 2022-03-05 10:47:12,294 WARN [Close-WAL-Writer-61] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477126836.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477126836.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 10:47:18,733 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 912 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:47:51,268 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df. because 80a1ffe1dc2d8f258a6367d204dc64df/e has an old edit so flush to free WALs after random delay 36096 ms 2022-03-05 10:48:01,349 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5. because 7534b3b1812808d2d3c16342fbf347a5/e has an old edit so flush to free WALs after random delay 232770 ms 2022-03-05 10:48:04,397 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1168 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:48:04,397 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 950 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:48:04,962 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 564 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:48:33,261 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 6600ms No GCs detected 2022-03-05 10:48:33,262 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 80a1ffe1dc2d8f258a6367d204dc64df 1/1 column families, dataSize=303.91 KB heapSize=569.92 KB 2022-03-05 10:48:33,289 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 7252 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:48:33,616 INFO [Thread-1237] retry.RetryInvocationHandler: at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:568) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.addBlock over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 10:48:33,633 INFO [Thread-1237] retry.RetryInvocationHandler: ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:568) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.addBlock over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 411ms. 2022-03-05 10:48:34,047 INFO [Thread-1237] retry.RetryInvocationHandler: lientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:568) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.addBlock over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 2 failover attempts. Trying to failover after sleeping for 1392ms. 2022-03-05 10:48:40,435 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=303.91 KB at sequenceid=9240 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/.tmp/e/5dcde96c4bf7443f997b4b496f352c2c 2022-03-05 10:48:40,453 INFO [MemStoreFlusher.1] retry.RetryInvocationHandler: doop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:433) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getBlockLocations over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 10:48:40,469 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/5dcde96c4bf7443f997b4b496f352c2c, entries=3631, sequenceid=9240, filesize=40.1 K 2022-03-05 10:48:40,474 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~303.91 KB/311202, heapSize ~569.91 KB/583584, currentSize=1.59 KB/1631 for 80a1ffe1dc2d8f258a6367d204dc64df in 7213ms, sequenceid=9240, compaction requested=false 2022-03-05 10:48:46,749 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 769 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:48:51,904 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5151 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:48:51,904 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4452 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:48:51,909 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2552ms No GCs detected 2022-03-05 10:48:55,779 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 835 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:48:57,006 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 999 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:48:57,393 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 901 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:48:59,719 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1308ms No GCs detected 2022-03-05 10:49:04,735 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2119ms No GCs detected 2022-03-05 10:49:05,040 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5318 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:49:12,078 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1249 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:49:12,083 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 580 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:49:12,978 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1062ms No GCs detected 2022-03-05 10:49:16,541 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4361 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:49:23,497 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:49:24,321 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:49:24,321 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:49:24,628 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477211445 with entries=339, filesize=148.47 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477364340 2022-03-05 10:49:24,628 WARN [Close-WAL-Writer-61] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:49:24,628 WARN [Close-WAL-Writer-61] wal.AsyncProtobufLogWriter: normal close failed, try recover java.lang.IllegalStateException: should call flush first before calling close at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:510) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:514) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:49:24,628 INFO [Close-WAL-Writer-61] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477211445 2022-03-05 10:49:24,629 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646477352997","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":11632,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":917,"class":"HRegionServer","multi.mutations":1} 2022-03-05 10:49:24,745 INFO [Close-WAL-Writer-61] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477211445 after 117ms 2022-03-05 10:49:28,978 INFO [Close-WAL-Writer-61] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477211445 after 4350ms 2022-03-05 10:49:29,161 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1237 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:49:29,161 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1022 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:49:29,161 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 953 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:49:29,161 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 695 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:49:30,945 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1608 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:49:36,080 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2772ms No GCs detected 2022-03-05 10:49:36,118 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6630 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:49:38,645 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 9119 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:49:38,646 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 8026 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:49:39,616 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 8894 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:49:39,616 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3497 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:49:39,616 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2724 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:49:39,616 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2149 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:49:39,616 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1986 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:49:39,617 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 662 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:49:41,561 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 685 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:49:41,761 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 10:49:41,761 INFO [Close-WAL-Writer-61] retry.RetryInvocationHandler: apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.isFileClosed(ClientNamenodeProtocolServerSideTranslatorPB.java:1322) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.isFileClosed over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 10:49:41,761 WARN [Close-WAL-Writer-61] retry.RetryInvocationHandler: A failover has occurred since the start of call #4267 ClientNamenodeProtocolTranslatorPB.isFileClosed over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 2022-03-05 10:49:41,770 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 1 failover attempts. Trying to failover after sleeping for 695ms. 2022-03-05 10:49:41,779 INFO [Close-WAL-Writer-61] retry.RetryInvocationHandler: enodeProtocolServerSideTranslatorPB.isFileClosed(ClientNamenodeProtocolServerSideTranslatorPB.java:1322) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.isFileClosed over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 1 failover attempts. Trying to failover after sleeping for 469ms. 2022-03-05 10:49:42,252 INFO [Close-WAL-Writer-61] retry.RetryInvocationHandler: nodeProtocolServerSideTranslatorPB.isFileClosed(ClientNamenodeProtocolServerSideTranslatorPB.java:1322) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.isFileClosed over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 2 failover attempts. Trying to failover after sleeping for 1201ms. 2022-03-05 10:49:42,468 WARN [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: A failover has occurred since the start of call #4266 ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 2022-03-05 10:49:42,519 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: tNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 2 failover attempts. Trying to failover after sleeping for 1080ms. 2022-03-05 10:49:44,067 WARN [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: A failover has occurred since the start of call #4266 ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 2022-03-05 10:49:46,521 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1306ms No GCs detected 2022-03-05 10:49:53,447 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:49:53,447 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:49:53,447 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:49:53,448 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:49:53,637 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 5941ms No GCs detected 2022-03-05 10:49:58,681 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1278ms No GCs detected 2022-03-05 10:50:01,019 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477228077.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477387747.meta 2022-03-05 10:50:01,032 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477228077.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477228077.meta 2022-03-05 10:50:01,020 WARN [Close-WAL-Writer-62] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:50:01,033 WARN [Close-WAL-Writer-62] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:50:01,033 INFO [Close-WAL-Writer-62] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477228077.meta 2022-03-05 10:50:01,075 INFO [Close-WAL-Writer-62] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477228077.meta after 42ms 2022-03-05 10:50:01,248 WARN [Close-WAL-Writer-61] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:50:01,248 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477364340 with entries=75, filesize=33.59 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477401032 2022-03-05 10:50:01,249 WARN [Close-WAL-Writer-61] wal.AsyncProtobufLogWriter: normal close failed, try recover java.lang.IllegalStateException: should call flush first before calling close at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:510) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:514) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:50:01,249 INFO [Close-WAL-Writer-61] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477364340 2022-03-05 10:50:01,270 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646477388230","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":13040,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":1709,"class":"HRegionServer","multi.mutations":1} 2022-03-05 10:50:01,270 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646477387747","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":13523,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":1226,"class":"HRegionServer","multi.mutations":1} 2022-03-05 10:50:01,271 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646477386521","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":14749,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 10:50:01,271 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646477387939","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":13332,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 10:50:01,495 INFO [Close-WAL-Writer-61] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477364340 after 246ms 2022-03-05 10:50:05,077 WARN [Close-WAL-Writer-62] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477228077.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477228077.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 10:50:05,497 INFO [Close-WAL-Writer-61] util.RecoverLeaseFSUtils: Recovered lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477364340 after 4248ms 2022-03-05 10:50:12,403 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 632 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:50:13,866 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1024 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:50:15,619 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1272ms No GCs detected 2022-03-05 10:50:20,416 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 10:50:20,495 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 843ms. 2022-03-05 10:50:20,520 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4917 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:50:20,520 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4914 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:50:22,185 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1662 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:50:23,224 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1168 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:50:23,224 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1026 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:50:31,191 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 7d1b53259f1d8280a47f7704f98f2c9f 1/1 column families, dataSize=672.36 KB heapSize=1.13 MB 2022-03-05 10:50:32,655 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. because 7d1b53259f1d8280a47f7704f98f2c9f/e has an old edit so flush to free WALs after random delay 294140 ms 2022-03-05 10:50:37,689 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 8552 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:50:37,689 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 8551 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:50:37,689 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 8550 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:50:37,689 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 7814 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:50:37,689 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 7133 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:50:37,689 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6847 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:50:37,689 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6511 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:50:39,171 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6319 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:50:39,171 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3016 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:50:39,171 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2289 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:50:39,171 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1482 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:50:39,171 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1481 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:50:49,290 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:50:49,588 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:50:49,588 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:50:49,588 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:50:49,588 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:50:49,588 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:50:49,588 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:50:49,588 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:50:49,588 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:50:49,588 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:50:54,756 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1712ms No GCs detected 2022-03-05 10:50:59,983 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:568) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.addBlock over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 10:50:59,988 INFO [Thread-1247] retry.RetryInvocationHandler: at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:568) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.addBlock over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 10:50:59,988 WARN [Thread-1247] retry.RetryInvocationHandler: A failover has occurred since the start of call #4288 ClientNamenodeProtocolTranslatorPB.addBlock over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 2022-03-05 10:51:00,006 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:568) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.addBlock over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 628ms. 2022-03-05 10:51:00,006 INFO [Thread-1247] retry.RetryInvocationHandler: ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:568) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.addBlock over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 362ms. 2022-03-05 10:51:00,433 INFO [Thread-1247] retry.RetryInvocationHandler: ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:568) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.addBlock over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 2 failover attempts. Trying to failover after sleeping for 805ms. 2022-03-05 10:51:00,634 WARN [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: A failover has occurred since the start of call #4287 ClientNamenodeProtocolTranslatorPB.addBlock over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 2022-03-05 10:51:00,794 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:568) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.addBlock over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 2 failover attempts. Trying to failover after sleeping for 997ms. 2022-03-05 10:51:01,791 WARN [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: A failover has occurred since the start of call #4287 ClientNamenodeProtocolTranslatorPB.addBlock over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 2022-03-05 10:51:07,552 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1272ms No GCs detected 2022-03-05 10:51:12,222 WARN [Close-WAL-Writer-62] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:51:12,222 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477401032 with entries=126, filesize=55.96 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477451125 2022-03-05 10:51:12,222 WARN [Close-WAL-Writer-62] wal.AsyncProtobufLogWriter: normal close failed, try recover java.lang.IllegalStateException: should call flush first before calling close at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:510) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:514) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:51:12,223 INFO [Close-WAL-Writer-62] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477401032 2022-03-05 10:51:12,225 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646477447595","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":24630,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":2,"class":"HRegionServer","multi.mutations":1} 2022-03-05 10:51:12,225 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646477446507","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":25718,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 10:51:12,244 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646477456019","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":16224,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 10:51:12,244 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646477440374","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":31869,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 10:51:12,244 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646477439342","responsesize":"16","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":32901,"client":"172.31.0.219:43118","multi.service_calls":0,"queuetimems":5,"class":"HRegionServer","multi.mutations":2} 2022-03-05 10:51:12,244 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646477444099","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":28145,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 10:51:12,245 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646477453579","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":18666,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 10:51:12,245 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646477455745","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":16500,"client":"172.30.41.155:52690","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 10:51:12,245 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646477452543","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":19702,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 10:51:12,245 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646477446508","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":25737,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-05 10:51:12,245 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646477441416","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":30829,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 10:51:12,245 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646477441878","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":30367,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 10:51:12,245 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646477446507","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":25738,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 10:51:12,245 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646477441956","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":30289,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 10:51:12,247 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646477454790","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":17457,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 10:51:12,250 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646477449081","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":23166,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 10:51:12,291 INFO [Close-WAL-Writer-62] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477401032 after 68ms 2022-03-05 10:51:12,368 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477387747.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477472223.meta 2022-03-05 10:51:12,368 WARN [Close-WAL-Writer-63] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:51:12,368 WARN [Close-WAL-Writer-63] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:51:12,368 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477387747.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477387747.meta 2022-03-05 10:51:12,375 INFO [Close-WAL-Writer-63] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477387747.meta 2022-03-05 10:51:12,423 INFO [Close-WAL-Writer-63] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477387747.meta after 48ms 2022-03-05 10:51:12,484 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=675.14 KB at sequenceid=8877 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/.tmp/e/4ff70de5921543e2ae6ff6c15e006596 2022-03-05 10:51:12,524 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/4ff70de5921543e2ae6ff6c15e006596, entries=6505, sequenceid=8877, filesize=64.7 K 2022-03-05 10:51:12,611 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~675.14 KB/691339, heapSize ~1.13 MB/1185464, currentSize=4.95 KB/5066 for 7d1b53259f1d8280a47f7704f98f2c9f in 41755ms, sequenceid=8877, compaction requested=false 2022-03-05 10:51:16,113 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1324 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:51:16,113 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1313 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:51:16,424 WARN [Close-WAL-Writer-63] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477387747.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477387747.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 10:51:16,602 INFO [Close-WAL-Writer-62] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477401032 after 4379ms 2022-03-05 10:51:17,105 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 992 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:51:17,511 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1354 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:51:30,582 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2477 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:51:41,473 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e. because 4333b489291b1f77671b8f2e78779a1e/__status_family___ has an old edit so flush to free WALs after random delay 80128 ms 2022-03-05 10:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=511.11 KB, freeSize=306.70 MB, max=307.20 MB, blockCount=21, accesses=6893, hits=6183, hitRatio=89.70%, , cachingAccesses=6222, cachingHits=6172, cachingHitsRatio=99.20%, evictions=2608, evicted=0, evictedPerRun=0.0 2022-03-05 10:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 10:51:54,119 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 7534b3b1812808d2d3c16342fbf347a5 1/1 column families, dataSize=186.83 KB heapSize=347.88 KB 2022-03-05 10:51:54,168 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=186.83 KB at sequenceid=7499 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/.tmp/e/0c7cf3c4005e4bf9af60a7450ca97afb 2022-03-05 10:51:54,185 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/0c7cf3c4005e4bf9af60a7450ca97afb, entries=2170, sequenceid=7499, filesize=26.0 K 2022-03-05 10:51:54,187 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~186.83 KB/191311, heapSize ~347.86 KB/356208, currentSize=0 B/0 for 7534b3b1812808d2d3c16342fbf347a5 in 68ms, sequenceid=7499, compaction requested=true 2022-03-05 10:51:54,187 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 7534b3b1812808d2d3c16342fbf347a5/e in aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5. 2022-03-05 10:51:54,187 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/dde113ca411d4939b7fcfe2f368dda2c, hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/f6d45b55baa640c6bce1a9192419bb27, hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/0c7cf3c4005e4bf9af60a7450ca97afb] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/.tmp, totalSize=207.5 K 2022-03-05 10:51:54,233 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 7534b3b1812808d2d3c16342fbf347a5#e#compaction#70 average throughput is 53.72 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 10:51:54,303 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 7534b3b1812808d2d3c16342fbf347a5/e of 7534b3b1812808d2d3c16342fbf347a5 into 503211fb70ab41168efb8399e86362bf(size=201.4 K), total size for store is 201.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 10:51:54,303 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., storeName=7534b3b1812808d2d3c16342fbf347a5/e, priority=13, startTime=1646477514187; duration=0sec 2022-03-05 10:52:04,285 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 524 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:52:22,688 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 9804 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:52:22,688 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 6016ms GC pool 'G1 Young Generation' had collection(s): count=1 time=6498ms 2022-03-05 10:52:26,510 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1235 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:52:26,512 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 541 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:52:36,126 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 509 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:52:50,798 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6013 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:52:50,798 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5957 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:52:50,798 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4730 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:52:50,957 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3407 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:52:50,957 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3092 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:52:53,179 WARN [regionserver/cvp328:16201.logRoller] asyncfs.FanOutOneBlockAsyncDFSOutputHelper: create fan-out dfs output /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477562685.meta failed, retry = 0 java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$3.userEventTriggered(FanOutOneBlockAsyncDFSOutputHelper.java:386) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:52:53,714 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2329 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:52:53,714 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1567 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:52:53,714 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 791 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:52:53,744 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477472223.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477562685.meta 2022-03-05 10:52:53,744 WARN [Close-WAL-Writer-64] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:52:53,744 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477472223.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477472223.meta 2022-03-05 10:52:53,744 WARN [Close-WAL-Writer-64] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:52:53,744 INFO [Close-WAL-Writer-64] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477472223.meta 2022-03-05 10:52:53,768 INFO [Close-WAL-Writer-64] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477472223.meta after 24ms 2022-03-05 10:52:57,771 WARN [Close-WAL-Writer-64] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477472223.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477472223.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 10:53:01,702 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1590ms No GCs detected 2022-03-05 10:53:01,705 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 4333b489291b1f77671b8f2e78779a1e 1/1 column families, dataSize=7.74 KB heapSize=12.36 KB 2022-03-05 10:53:05,384 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb. because ccdacf4b414cb8d1bc548efc77c663cb/e has an old edit so flush to free WALs after random delay 190814 ms 2022-03-05 10:53:05,490 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3288ms No GCs detected 2022-03-05 10:53:05,903 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4200 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:53:05,903 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4198 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:53:05,903 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4197 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:53:05,903 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4196 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:53:05,903 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4192 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:53:07,590 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1555 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:53:07,590 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1486 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:53:07,590 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1485 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:53:08,269 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=7.74 KB at sequenceid=526 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/.tmp/__status_family___/9fd38565966a4d6a9466447681a255c9 2022-03-05 10:53:08,280 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/9fd38565966a4d6a9466447681a255c9, entries=3, sequenceid=526, filesize=5.1 K 2022-03-05 10:53:08,281 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~7.74 KB/7924, heapSize ~12.34 KB/12640, currentSize=0 B/0 for 4333b489291b1f77671b8f2e78779a1e in 6576ms, sequenceid=526, compaction requested=true 2022-03-05 10:53:08,282 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 4333b489291b1f77671b8f2e78779a1e/__status_family___ in __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e. 2022-03-05 10:53:08,282 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/f76c2bd644a34f6284d6c442ed66fbaf, hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/69725697445d4cac8df314f18a8d0f41, hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/9fd38565966a4d6a9466447681a255c9] into tmpdir=hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/.tmp, totalSize=15.4 K 2022-03-05 10:53:08,295 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 4333b489291b1f77671b8f2e78779a1e#__status_family___#compaction#72 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 10:53:08,381 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 4333b489291b1f77671b8f2e78779a1e/__status_family___ of 4333b489291b1f77671b8f2e78779a1e into 6e6329e0b80945efa437da601d441ae6(size=5.2 K), total size for store is 5.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 10:53:08,381 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=__test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., storeName=4333b489291b1f77671b8f2e78779a1e/__status_family___, priority=13, startTime=1646477588282; duration=0sec 2022-03-05 10:53:34,861 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1973 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:53:34,886 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1174 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:53:34,886 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1074 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:53:34,886 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 672 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:54:58,431 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2936ms No GCs detected 2022-03-05 10:55:26,796 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 7d1b53259f1d8280a47f7704f98f2c9f 1/1 column families, dataSize=49.96 KB heapSize=86.04 KB 2022-03-05 10:55:26,803 INFO [MemStoreFlusher.0] retry.RetryInvocationHandler: rg.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 10:55:26,867 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=49.96 KB at sequenceid=8969 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/.tmp/e/cf34be11109743918fba4c47bdcf6598 2022-03-05 10:55:26,873 INFO [MemStoreFlusher.0] retry.RetryInvocationHandler: doop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:433) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getBlockLocations over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 10:55:26,882 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/cf34be11109743918fba4c47bdcf6598, entries=483, sequenceid=8969, filesize=9.5 K 2022-03-05 10:55:26,883 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~49.96 KB/51157, heapSize ~86.02 KB/88088, currentSize=0 B/0 for 7d1b53259f1d8280a47f7704f98f2c9f in 87ms, sequenceid=8969, compaction requested=true 2022-03-05 10:55:26,884 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 7d1b53259f1d8280a47f7704f98f2c9f/e in aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. 2022-03-05 10:55:26,884 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/0be95bb2a69b403aa8bcf36844017561, hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/4ff70de5921543e2ae6ff6c15e006596, hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/cf34be11109743918fba4c47bdcf6598] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/.tmp, totalSize=623.2 K 2022-03-05 10:55:26,995 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 7d1b53259f1d8280a47f7704f98f2c9f#e#compaction#74 average throughput is 52.00 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 10:55:27,197 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 7d1b53259f1d8280a47f7704f98f2c9f/e of 7d1b53259f1d8280a47f7704f98f2c9f into d7067341838449ddaf582a633d1a2665(size=638.9 K), total size for store is 638.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 10:55:27,197 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., storeName=7d1b53259f1d8280a47f7704f98f2c9f/e, priority=13, startTime=1646477726883; duration=0sec 2022-03-05 10:56:09,031 INFO [regionserver/cvp328:16201.Chore.3] hbase.ScheduledChore: Chore: CompactionChecker missed its start time 2022-03-05 10:56:09,031 INFO [regionserver/cvp328:16201.Chore.3] hbase.ScheduledChore: Chore: MemstoreFlusherChore missed its start time 2022-03-05 10:56:09,048 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 12220 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:56:09,048 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Requesting log roll because we exceeded slow sync threshold; time=12220 ms, threshold=10000 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:56:09,048 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 11418 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:56:09,048 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Requesting log roll because we exceeded slow sync threshold; time=11418 ms, threshold=10000 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:56:09,048 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 11110 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:56:09,048 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Requesting log roll because we exceeded slow sync threshold; time=11110 ms, threshold=10000 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:56:09,048 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 10984 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:56:09,048 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Requesting log roll because we exceeded slow sync threshold; time=10984 ms, threshold=10000 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:56:09,049 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646477756828","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":12220,"client":"172.30.41.155:52690","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 10:56:09,049 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.sync(AsyncProtobufLogWriter.java:148) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:394) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.consume(AsyncFSWAL.java:581) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:56:09,049 WARN [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 10414ms GC pool 'G1 Young Generation' had collection(s): count=1 time=320ms 2022-03-05 10:56:09,050 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646477757937","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":11112,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 10:56:09,050 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646477757629","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":11421,"client":"172.30.41.155:52690","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 10:56:09,062 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646477758064","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":10998,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 10:56:09,326 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 10:56:09,326 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: rg.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 10:56:09,326 WARN [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: A failover has occurred since the start of call #4524 ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 2022-03-05 10:56:09,335 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 444ms. 2022-03-05 10:56:09,340 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: NamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 405ms. 2022-03-05 10:56:09,760 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: amenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 2 failover attempts. Trying to failover after sleeping for 1226ms. 2022-03-05 10:56:09,779 WARN [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: A failover has occurred since the start of call #4523 ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 2022-03-05 10:56:09,845 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: tNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 2 failover attempts. Trying to failover after sleeping for 1381ms. 2022-03-05 10:56:11,226 WARN [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: A failover has occurred since the start of call #4523 ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 2022-03-05 10:56:13,261 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477451125 with entries=717, filesize=304.03 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477769048 2022-03-05 10:56:13,261 WARN [Close-WAL-Writer-63] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:56:13,261 WARN [Close-WAL-Writer-63] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:56:13,262 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646473286132 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646473286132 2022-03-05 10:56:13,262 INFO [Close-WAL-Writer-63] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477451125 2022-03-05 10:56:13,828 INFO [Close-WAL-Writer-63] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477451125 after 566ms 2022-03-05 10:56:23,345 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing ccdacf4b414cb8d1bc548efc77c663cb 1/1 column families, dataSize=296.90 KB heapSize=513.84 KB 2022-03-05 10:56:25,945 INFO [regionserver/cvp328:16201.Chore.1] hbase.ScheduledChore: Chore: CompactionChecker missed its start time 2022-03-05 10:56:27,854 WARN [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 13530ms No GCs detected 2022-03-05 10:56:27,927 INFO [regionserver/cvp328:16201.Chore.3] hbase.ScheduledChore: Chore: MemstoreFlusherChore missed its start time 2022-03-05 10:56:28,206 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.sync(AsyncProtobufLogWriter.java:148) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:394) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.consume(AsyncFSWAL.java:581) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:56:30,084 INFO [Close-WAL-Writer-63] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477451125 after 16822ms 2022-03-05 10:56:30,112 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477562685.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477787917.meta 2022-03-05 10:56:30,112 WARN [Close-WAL-Writer-65] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:56:30,113 WARN [Close-WAL-Writer-65] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:56:30,113 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477562685.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477562685.meta 2022-03-05 10:56:30,113 INFO [Close-WAL-Writer-65] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477562685.meta 2022-03-05 10:56:30,144 INFO [Close-WAL-Writer-65] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477562685.meta after 31ms 2022-03-05 10:56:30,185 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477769048 with entries=62, filesize=27.71 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477790113 2022-03-05 10:56:30,185 WARN [Close-WAL-Writer-64] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:56:30,185 WARN [Close-WAL-Writer-64] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:56:30,185 INFO [Close-WAL-Writer-64] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477769048 2022-03-05 10:56:30,214 INFO [Close-WAL-Writer-64] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477769048 after 28ms 2022-03-05 10:56:30,321 INFO [MemStoreFlusher.1] retry.RetryInvocationHandler: at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:614) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.complete over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 10:56:30,329 INFO [MemStoreFlusher.1] retry.RetryInvocationHandler: ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:614) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.complete over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 1 failover attempts. Trying to failover after sleeping for 883ms. 2022-03-05 10:56:31,214 INFO [MemStoreFlusher.1] retry.RetryInvocationHandler: lientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:614) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.complete over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 2 failover attempts. Trying to failover after sleeping for 1504ms. 2022-03-05 10:56:33,439 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=296.90 KB at sequenceid=5101 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/.tmp/e/0455b4138e6048bd9e1337307e723232 2022-03-05 10:56:33,792 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/0455b4138e6048bd9e1337307e723232, entries=2943, sequenceid=5101, filesize=30.9 K 2022-03-05 10:56:33,814 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~296.90 KB/304028, heapSize ~513.82 KB/526152, currentSize=2.35 KB/2403 for ccdacf4b414cb8d1bc548efc77c663cb in 10469ms, sequenceid=5101, compaction requested=false 2022-03-05 10:56:34,152 WARN [Close-WAL-Writer-65] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477562685.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477562685.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 10:56:34,276 INFO [Close-WAL-Writer-64] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477769048 after 4090ms 2022-03-05 10:56:39,719 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 606 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=391.97 KB, freeSize=306.82 MB, max=307.20 MB, blockCount=14, accesses=7025, hits=6209, hitRatio=88.38%, , cachingAccesses=6244, cachingHits=6194, cachingHitsRatio=99.20%, evictions=2637, evicted=0, evictedPerRun=0.0 2022-03-05 10:56:47,077 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1224 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:56:51,660 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5539 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:56:51,690 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3029ms No GCs detected 2022-03-05 10:56:51,758 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 10:57:02,393 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 974 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 10:57:19,943 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391. because 0a764e51258807790269e46c55b99391/e has an old edit so flush to free WALs after random delay 278892 ms 2022-03-05 10:57:22,595 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 5778ms No GCs detected 2022-03-05 10:57:23,773 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:57:26,684 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:57:26,684 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:57:26,684 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:57:26,684 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.sync(AsyncProtobufLogWriter.java:148) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:394) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.consume(AsyncFSWAL.java:581) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:57:28,984 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: rg.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 10:57:29,257 WARN [Close-WAL-Writer-64] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:57:29,258 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477790113 with entries=142, filesize=64.42 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477846685 2022-03-05 10:57:29,258 WARN [Close-WAL-Writer-64] wal.AsyncProtobufLogWriter: normal close failed, try recover java.lang.IllegalStateException: should call flush first before calling close at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:510) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:514) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:57:29,258 INFO [Close-WAL-Writer-64] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477790113 2022-03-05 10:57:29,263 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646477834965","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":14298,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-05 10:57:29,263 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646477834964","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":14299,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 10:57:29,264 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646477836813","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":12450,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 10:57:29,360 INFO [Close-WAL-Writer-64] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477790113 after 102ms 2022-03-05 10:57:33,361 INFO [Close-WAL-Writer-64] util.RecoverLeaseFSUtils: Recovered lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477790113 after 4103ms 2022-03-05 10:57:36,768 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1164 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:57:39,725 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477787917.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477859258.meta 2022-03-05 10:57:39,726 WARN [Close-WAL-Writer-66] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:57:39,726 WARN [Close-WAL-Writer-66] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 10:57:39,726 INFO [Close-WAL-Writer-66] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477787917.meta 2022-03-05 10:57:39,726 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477787917.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477787917.meta 2022-03-05 10:57:39,734 INFO [Close-WAL-Writer-66] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477787917.meta after 8ms 2022-03-05 10:57:43,736 WARN [Close-WAL-Writer-66] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477787917.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477787917.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 10:58:25,652 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 896 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:58:33,932 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1202 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:58:33,932 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1104 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:58:44,359 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3968 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:58:44,359 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2972 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:58:44,359 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2586 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:58:44,359 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2528 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:58:44,359 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1035 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:58:56,661 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1880 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:59:02,989 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 868 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:59:02,989 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 868 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:59:06,342 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2149 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:59:06,342 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2146 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:59:06,342 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2141 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:59:06,979 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 636 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:59:31,181 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2732 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:59:31,187 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1779 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:59:31,187 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1679 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:59:31,187 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 660 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:59:31,187 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 630 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 10:59:42,392 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 576 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:00:06,355 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1809ms No GCs detected 2022-03-05 11:00:06,424 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2837 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:00:09,575 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1373 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:00:12,628 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 529 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:00:18,591 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4195 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:00:18,591 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3465 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:00:18,591 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2999 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:00:18,591 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1323 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:00:18,591 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1002 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:00:18,591 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 677 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:00:19,376 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 663 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:00:33,003 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 8231 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:00:33,003 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6114 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:00:33,003 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6112 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:00:33,003 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6108 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:00:33,003 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5523 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:00:33,004 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4925 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:00:33,004 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4249 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:00:33,004 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4246 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:00:33,004 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3569 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:00:33,004 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2973 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:00:33,004 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2966 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:00:33,004 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2228 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:00:33,004 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2137 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:00:34,129 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 608 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:00:35,144 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 927 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:00:38,472 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3326 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:00:38,472 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2707 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:00:38,472 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 844 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:01:01,515 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2055 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:01:01,515 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2032 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:01:01,515 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1205 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:01:01,515 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 919 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:01:01,515 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 718 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:01:01,515 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 539 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:01:13,932 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1049 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=391.97 KB, freeSize=306.82 MB, max=307.20 MB, blockCount=14, accesses=7047, hits=6231, hitRatio=88.42%, , cachingAccesses=6266, cachingHits=6216, cachingHitsRatio=99.20%, evictions=2666, evicted=0, evictedPerRun=0.0 2022-03-05 11:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 11:01:48,668 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1209 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:01:48,669 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 895 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:01:56,738 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 687 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:01:58,840 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 0a764e51258807790269e46c55b99391 1/1 column families, dataSize=968.56 KB heapSize=1.70 MB 2022-03-05 11:01:59,926 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=968.56 KB at sequenceid=17333 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/.tmp/e/b6dc1513470848399c60dc92add59fd4 2022-03-05 11:01:59,932 INFO [MemStoreFlusher.0] retry.RetryInvocationHandler: doop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:433) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getBlockLocations over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 11:01:59,944 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/b6dc1513470848399c60dc92add59fd4, entries=10394, sequenceid=17333, filesize=93.7 K 2022-03-05 11:01:59,945 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~968.56 KB/991805, heapSize ~1.70 MB/1780480, currentSize=390 B/390 for 0a764e51258807790269e46c55b99391 in 1105ms, sequenceid=17333, compaction requested=false 2022-03-05 11:02:32,123 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1724 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:02:32,123 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1274 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:02:32,123 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1124 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:02:32,124 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 880 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:02:43,502 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1056ms No GCs detected 2022-03-05 11:02:43,839 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1895 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:02:50,250 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 11:03:30,987 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 616 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:03:31,612 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 622 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:04:27,353 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2539 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:04:48,002 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 558 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:05:13,417 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1020ms No GCs detected 2022-03-05 11:05:14,773 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 943 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:05:46,772 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 665 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:05:47,361 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 567 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:06:00,297 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2241 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:06:00,412 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2356 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:06:00,412 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2255 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:06:00,412 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1947 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:06:00,412 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1702 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:06:06,264 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 679 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:06:12,616 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1831 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:06:12,616 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1335 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:06:12,616 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 759 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=391.97 KB, freeSize=306.82 MB, max=307.20 MB, blockCount=14, accesses=7069, hits=6253, hitRatio=88.46%, , cachingAccesses=6288, cachingHits=6238, cachingHitsRatio=99.20%, evictions=2696, evicted=0, evictedPerRun=0.0 2022-03-05 11:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 11:06:55,967 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 801 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:07:07,460 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1887 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:07:07,460 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1246 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:07:27,564 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 11:07:32,243 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477846685 with entries=1353, filesize=600.99 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646478452223 2022-03-05 11:07:32,243 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646473739902 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646473739902 2022-03-05 11:07:32,247 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646474345455 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646474345455 2022-03-05 11:07:32,251 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646474495542 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646474495542 2022-03-05 11:07:32,254 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646474656245 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646474656245 2022-03-05 11:07:32,261 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646475259275 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646475259275 2022-03-05 11:07:32,264 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646475719936 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646475719936 2022-03-05 11:07:32,269 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646476325696 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646476325696 2022-03-05 11:07:32,272 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646476613242 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646476613242 2022-03-05 11:07:43,416 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477859258.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646478462243.meta 2022-03-05 11:07:43,416 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477859258.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646477859258.meta 2022-03-05 11:07:53,079 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 874 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:08:01,357 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 610 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:08:10,045 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1648 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:08:14,841 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1951 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:08:28,138 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 637 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:08:28,698 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 587 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:08:29,846 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1374 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:08:29,868 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1170 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:09:03,096 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3756ms No GCs detected 2022-03-05 11:09:04,714 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6582 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:09:05,806 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 696 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:09:15,813 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: rg.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 11:09:16,160 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1183 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:09:16,350 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646478462243.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646478554728.meta 2022-03-05 11:09:16,350 WARN [Close-WAL-Writer-68] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 11:09:16,350 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646478462243.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646478462243.meta 2022-03-05 11:09:16,350 WARN [Close-WAL-Writer-68] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 11:09:16,350 INFO [Close-WAL-Writer-68] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646478462243.meta 2022-03-05 11:09:16,383 INFO [Close-WAL-Writer-68] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646478462243.meta after 33ms 2022-03-05 11:09:20,385 WARN [Close-WAL-Writer-68] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646478462243.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646478462243.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 11:09:25,266 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1971 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:09:25,266 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 995 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:09:25,267 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 695 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:09:34,124 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 844 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:09:40,759 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4101ms No GCs detected 2022-03-05 11:09:40,794 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6655 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:09:48,953 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3466 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:09:48,953 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2573 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:09:48,953 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1562 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:09:49,270 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1432 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:09:52,958 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 892 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:09:57,380 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1029ms No GCs detected 2022-03-05 11:10:01,912 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 8954 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:10:01,938 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 8648 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:10:01,938 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 7068 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:10:01,938 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4414 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:10:01,938 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4404 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:10:11,830 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6127 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:10:11,830 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2067 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:10:19,766 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 11:10:22,476 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 11:10:22,476 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 11:10:22,477 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 11:10:22,477 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 11:10:22,477 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 11:10:22,477 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 11:10:22,477 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 11:10:24,752 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646478554728.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646478622234.meta 2022-03-05 11:10:24,752 WARN [Close-WAL-Writer-69] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 11:10:24,754 WARN [Close-WAL-Writer-69] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 11:10:24,754 INFO [Close-WAL-Writer-69] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646478554728.meta 2022-03-05 11:10:24,757 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646478554728.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646478554728.meta 2022-03-05 11:10:24,863 INFO [Close-WAL-Writer-69] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646478554728.meta after 109ms 2022-03-05 11:10:24,970 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:568) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.addBlock over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 11:10:24,970 INFO [Log-Archiver-0] retry.RetryInvocationHandler: 040) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.rename(ClientNamenodeProtocolServerSideTranslatorPB.java:652) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.rename over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 11:10:24,970 WARN [Log-Archiver-0] retry.RetryInvocationHandler: A failover has occurred since the start of call #4669 ClientNamenodeProtocolTranslatorPB.rename over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 2022-03-05 11:10:24,999 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:568) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.addBlock over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 331ms. 2022-03-05 11:10:25,000 INFO [Log-Archiver-0] retry.RetryInvocationHandler: lPB.ClientNamenodeProtocolServerSideTranslatorPB.rename(ClientNamenodeProtocolServerSideTranslatorPB.java:652) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.rename over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 376ms. 2022-03-05 11:10:25,349 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: lientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:568) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.addBlock over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 2 failover attempts. Trying to failover after sleeping for 1424ms. 2022-03-05 11:10:25,380 WARN [Log-Archiver-0] retry.RetryInvocationHandler: A failover has occurred since the start of call #4669 ClientNamenodeProtocolTranslatorPB.rename over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 2022-03-05 11:10:25,392 INFO [Log-Archiver-0] retry.RetryInvocationHandler: PB.ClientNamenodeProtocolServerSideTranslatorPB.rename(ClientNamenodeProtocolServerSideTranslatorPB.java:652) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.rename over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 2 failover attempts. Trying to failover after sleeping for 1143ms. 2022-03-05 11:10:26,773 WARN [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: A failover has occurred since the start of call #4668 ClientNamenodeProtocolTranslatorPB.addBlock over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 2022-03-05 11:10:27,819 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646478452223 with entries=345, filesize=142.77 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646478624752 2022-03-05 11:10:27,819 WARN [Close-WAL-Writer-66] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 11:10:27,819 WARN [Close-WAL-Writer-66] wal.AsyncProtobufLogWriter: normal close failed, try recover java.lang.IllegalStateException: should call flush first before calling close at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:510) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:514) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 11:10:27,819 INFO [Close-WAL-Writer-66] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646478452223 2022-03-05 11:10:27,822 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646478611851","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":15971,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 11:10:27,822 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646478611830","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":15992,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 11:10:27,823 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646478611830","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":15993,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 11:10:27,822 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646478611623","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":16199,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 11:10:27,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646478611971","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":15852,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":141,"class":"HRegionServer","multi.mutations":1} 2022-03-05 11:10:27,823 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646478612831","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":14992,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 11:10:27,823 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646478614794","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":13029,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 11:10:27,823 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646478616480","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":11343,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 11:10:27,823 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646478616871","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":10952,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 11:10:27,823 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646478615408","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":12415,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":624,"class":"HRegionServer","multi.mutations":1} 2022-03-05 11:10:27,833 INFO [Close-WAL-Writer-66] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646478452223 after 14ms 2022-03-05 11:10:28,864 WARN [Close-WAL-Writer-69] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646478554728.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646478554728.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 11:10:31,834 INFO [Close-WAL-Writer-66] util.RecoverLeaseFSUtils: Recovered lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646478452223 after 4015ms 2022-03-05 11:10:45,767 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1052 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:10:45,767 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 948 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:10:49,961 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2168 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:10:56,483 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1100ms No GCs detected 2022-03-05 11:11:15,998 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1207 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:11:16,060 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1255 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:11:29,063 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 643 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:11:29,063 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 643 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:11:42,341 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 578 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:11:42,341 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 539 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=391.97 KB, freeSize=306.82 MB, max=307.20 MB, blockCount=14, accesses=7091, hits=6275, hitRatio=88.49%, , cachingAccesses=6310, cachingHits=6260, cachingHitsRatio=99.21%, evictions=2726, evicted=0, evictedPerRun=0.0 2022-03-05 11:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 11:12:16,802 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1653ms No GCs detected 2022-03-05 11:12:24,835 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 11:12:29,902 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1477 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:12:47,117 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1016 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:12:58,856 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 788 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:13:04,042 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1111 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:13:14,081 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1838ms No GCs detected 2022-03-05 11:13:14,091 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3521 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:13:17,141 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1614 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:13:17,141 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 800 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:13:35,707 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1479ms No GCs detected 2022-03-05 11:13:36,603 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3556 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:13:36,603 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3379 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:13:48,582 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 897 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:13:59,696 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4922 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:13:59,696 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4330 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:13:59,696 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3681 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:13:59,696 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3634 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:13:59,792 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3312 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:13:59,792 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1980 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:13:59,792 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1871 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:14:13,095 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1205 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:14:13,103 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1206 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:14:26,106 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 11:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=391.97 KB, freeSize=306.82 MB, max=307.20 MB, blockCount=14, accesses=7113, hits=6297, hitRatio=88.53%, , cachingAccesses=6332, cachingHits=6282, cachingHitsRatio=99.21%, evictions=2755, evicted=0, evictedPerRun=0.0 2022-03-05 11:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 11:18:12,459 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 933 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:18:12,459 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 638 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:18:12,459 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 636 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:18:12,459 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 633 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:18:23,978 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2245ms No GCs detected 2022-03-05 11:18:26,975 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 11:18:34,254 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1459ms No GCs detected 2022-03-05 11:18:36,158 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5408 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:18:36,202 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5136 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:18:36,202 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5135 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:18:36,167 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1412ms No GCs detected 2022-03-05 11:18:37,628 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 973 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:18:38,433 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 804 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:18:38,433 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 777 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:18:57,437 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 11:19:13,434 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1552ms No GCs detected 2022-03-05 11:19:58,286 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1329ms No GCs detected 2022-03-05 11:19:58,876 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4874 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:19:58,876 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4586 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:19:59,294 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 11:20:07,750 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3636 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:20:07,947 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1522ms No GCs detected 2022-03-05 11:20:21,599 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2045ms No GCs detected 2022-03-05 11:20:21,600 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3731 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:20:21,600 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3201 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:20:24,578 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1542 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:20:24,578 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1209 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:20:24,672 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: rg.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 11:20:24,836 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646478622234.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646479224541.meta 2022-03-05 11:20:24,837 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646478622234.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646478622234.meta 2022-03-05 11:20:25,001 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646478624752 with entries=1339, filesize=586.02 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646479224837 2022-03-05 11:20:41,431 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4843ms No GCs detected 2022-03-05 11:20:53,858 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1421ms No GCs detected 2022-03-05 11:20:55,978 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 11:20:56,016 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1157ms No GCs detected 2022-03-05 11:20:56,084 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 11:20:56,084 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 11:20:56,084 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 11:20:56,084 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-05 11:21:05,421 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1021ms No GCs detected 2022-03-05 11:21:07,911 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646479224837 with entries=61, filesize=26.94 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646479256085 2022-03-05 11:21:07,911 WARN [Close-WAL-Writer-67] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 11:21:07,911 WARN [Close-WAL-Writer-67] wal.AsyncProtobufLogWriter: normal close failed, try recover java.lang.IllegalStateException: should call flush first before calling close at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:510) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:514) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 11:21:07,912 INFO [Close-WAL-Writer-67] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646479224837 2022-03-05 11:21:07,920 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646479246144","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":21776,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 11:21:07,920 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646479247451","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":20469,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 11:21:07,920 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646479247832","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":20088,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-05 11:21:07,921 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646479252127","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":15793,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 11:21:07,921 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646479256497","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":11424,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 11:21:07,922 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646479253766","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":14155,"client":"172.30.41.155:52690","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 11:21:07,922 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646479256017","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":11905,"client":"172.30.41.155:52690","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 11:21:07,922 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646479257277","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":10645,"client":"172.30.41.155:52690","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 11:21:07,922 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646479257902","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":10020,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 11:21:07,939 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646479256084","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":11855,"client":"172.30.41.155:52690","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 11:21:07,939 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646479256017","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":11922,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-05 11:21:07,939 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646479256098","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":11841,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":13,"class":"HRegionServer","multi.mutations":1} 2022-03-05 11:21:07,945 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646479257575","responsesize":"18","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":10370,"client":"172.30.41.155:52690","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-05 11:21:07,947 INFO [Close-WAL-Writer-67] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646479224837 after 35ms 2022-03-05 11:21:08,021 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646479224541.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646479267911.meta 2022-03-05 11:21:08,024 WARN [Close-WAL-Writer-70] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 11:21:08,025 WARN [Close-WAL-Writer-70] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 11:21:08,025 INFO [Close-WAL-Writer-70] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646479224541.meta 2022-03-05 11:21:08,032 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646479224541.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646479224541.meta 2022-03-05 11:21:08,061 INFO [Close-WAL-Writer-70] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646479224541.meta after 36ms 2022-03-05 11:21:12,025 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 511 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:21:12,028 INFO [Close-WAL-Writer-67] util.RecoverLeaseFSUtils: Recovered lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646479224837 after 4116ms 2022-03-05 11:21:13,508 WARN [Close-WAL-Writer-70] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646479224541.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646479224541.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 11:21:16,580 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2020ms No GCs detected 2022-03-05 11:21:18,985 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1404ms No GCs detected 2022-03-05 11:21:19,264 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5752 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:21:28,665 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3909 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:21:28,697 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2210ms No GCs detected 2022-03-05 11:21:30,975 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1422 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:21:30,976 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 840 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:21:37,974 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 11:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=391.97 KB, freeSize=306.82 MB, max=307.20 MB, blockCount=14, accesses=7135, hits=6319, hitRatio=88.56%, , cachingAccesses=6354, cachingHits=6304, cachingHitsRatio=99.21%, evictions=2785, evicted=0, evictedPerRun=0.0 2022-03-05 11:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 11:22:04,883 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 768 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:22:12,073 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 767 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:22:22,975 INFO [regionserver/cvp328:16201.Chore.3] hbase.ScheduledChore: Chore: MemstoreFlusherChore missed its start time 2022-03-05 11:22:23,011 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 5172ms No GCs detected 2022-03-05 11:22:23,951 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 936 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:22:23,951 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 934 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:22:24,868 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 917 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:22:26,639 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1809 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:22:26,639 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1111 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:22:38,630 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 11:23:30,944 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3687 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:23:30,944 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3407 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:23:30,944 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3079 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:23:35,684 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 7385 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:23:35,726 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 7426 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:23:35,726 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 7246 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:23:35,726 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6881 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:23:35,726 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6346 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:23:35,726 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6346 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:23:35,726 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6234 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:23:35,727 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5549 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:23:35,727 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5262 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:23:35,727 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4779 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:23:35,727 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4723 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:23:35,727 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4677 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:23:35,727 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4616 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:23:39,068 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 11:23:43,077 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1034ms No GCs detected 2022-03-05 11:23:43,080 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1328 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:23:43,534 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646479267911.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646479423511.meta 2022-03-05 11:23:43,534 WARN [Close-WAL-Writer-71] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 11:23:43,534 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646479267911.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646479267911.meta 2022-03-05 11:23:43,534 WARN [Close-WAL-Writer-71] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-05 11:23:43,535 INFO [Close-WAL-Writer-71] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646479267911.meta 2022-03-05 11:23:43,545 INFO [Close-WAL-Writer-71] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646479267911.meta after 10ms 2022-03-05 11:23:47,547 WARN [Close-WAL-Writer-71] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646479267911.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646479267911.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-05 11:24:11,180 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2130 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:24:17,285 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1518 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:24:17,285 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 828 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:26:23,385 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 503 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=391.97 KB, freeSize=306.82 MB, max=307.20 MB, blockCount=14, accesses=7157, hits=6341, hitRatio=88.60%, , cachingAccesses=6376, cachingHits=6326, cachingHitsRatio=99.22%, evictions=2815, evicted=0, evictedPerRun=0.0 2022-03-05 11:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 11:28:06,628 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 817 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:29:26,414 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1587 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:31:03,573 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646479256085 with entries=1356, filesize=582.77 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646479863560 2022-03-05 11:31:44,671 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-05 11:31:44,967 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=391.97 KB, freeSize=306.82 MB, max=307.20 MB, blockCount=14, accesses=7179, hits=6363, hitRatio=88.63%, , cachingAccesses=6398, cachingHits=6348, cachingHitsRatio=99.22%, evictions=2845, evicted=0, evictedPerRun=0.0 2022-03-05 11:31:45,003 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2118 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 11:32:29,522 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3486ms No GCs detected 2022-03-05 11:32:29,684 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4854 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:32:29,685 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4802 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:32:31,795 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1271ms No GCs detected 2022-03-05 11:32:31,795 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2690 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:32:31,795 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2689 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:32:31,795 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2336 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:32:31,795 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2326 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:32:31,795 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2308 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:32:31,796 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2111 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:32:31,796 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2036 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:32:31,796 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 510 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:32:43,422 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 11:33:43,650 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646479423511.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646480023637.meta 2022-03-05 11:33:43,650 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646479423511.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646479423511.meta 2022-03-05 11:35:10,950 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2958 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:35:10,952 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1703 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:35:10,952 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 632 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:35:43,693 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 11:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=391.97 KB, freeSize=306.82 MB, max=307.20 MB, blockCount=14, accesses=7207, hits=6391, hitRatio=88.68%, , cachingAccesses=6426, cachingHits=6376, cachingHitsRatio=99.22%, evictions=2875, evicted=0, evictedPerRun=0.0 2022-03-05 11:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 11:40:09,119 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3481 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:40:10,855 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2468 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:40:10,855 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2448 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:40:10,855 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1757 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 11:41:03,671 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646479863560 with entries=1303, filesize=585.69 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646480463655 2022-03-05 11:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=391.97 KB, freeSize=306.82 MB, max=307.20 MB, blockCount=14, accesses=7229, hits=6413, hitRatio=88.71%, , cachingAccesses=6448, cachingHits=6398, cachingHitsRatio=99.22%, evictions=2905, evicted=0, evictedPerRun=0.0 2022-03-05 11:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 11:43:43,690 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646480023637.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646480623673.meta 2022-03-05 11:43:43,691 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646480023637.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646480023637.meta 2022-03-05 11:44:22,594 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 537 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:44:24,264 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1381 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:44:24,264 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1155 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:44:35,793 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2944 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:44:35,793 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2943 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:44:35,793 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2666 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:45:43,279 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb. because 9b121c22e18472616debd0996ee2a4cb/e has an old edit so flush to free WALs after random delay 63500 ms 2022-03-05 11:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=391.97 KB, freeSize=306.82 MB, max=307.20 MB, blockCount=14, accesses=7251, hits=6435, hitRatio=88.75%, , cachingAccesses=6470, cachingHits=6420, cachingHitsRatio=99.23%, evictions=2935, evicted=0, evictedPerRun=0.0 2022-03-05 11:46:46,780 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 9b121c22e18472616debd0996ee2a4cb 1/1 column families, dataSize=553.84 KB heapSize=1023.02 KB 2022-03-05 11:46:46,926 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=553.84 KB at sequenceid=14340 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/.tmp/e/20a998c5e6d74218ba05152edf3d0b49 2022-03-05 11:46:46,932 INFO [MemStoreFlusher.1] retry.RetryInvocationHandler: doop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:433) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getBlockLocations over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 11:46:46,940 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/20a998c5e6d74218ba05152edf3d0b49, entries=6402, sequenceid=14340, filesize=67.1 K 2022-03-05 11:46:46,941 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~553.84 KB/567128, heapSize ~1023 KB/1047552, currentSize=0 B/0 for 9b121c22e18472616debd0996ee2a4cb in 161ms, sequenceid=14340, compaction requested=true 2022-03-05 11:46:46,942 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 9b121c22e18472616debd0996ee2a4cb/e in aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb. 2022-03-05 11:46:46,942 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/4f32e90c300b4130a465533983e52667, hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/66df821cd4fe4e3aa1f21097c8efd58b, hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/20a998c5e6d74218ba05152edf3d0b49] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/.tmp, totalSize=644.1 K 2022-03-05 11:46:47,064 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 9b121c22e18472616debd0996ee2a4cb#e#compaction#78 average throughput is 46.63 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 11:46:47,224 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 9b121c22e18472616debd0996ee2a4cb/e of 9b121c22e18472616debd0996ee2a4cb into 81bc2237797d4256b2edf2d91192e745(size=655.5 K), total size for store is 655.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 11:46:47,224 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., storeName=9b121c22e18472616debd0996ee2a4cb/e, priority=13, startTime=1646480806941; duration=0sec 2022-03-05 11:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 11:48:42,976 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df. because 80a1ffe1dc2d8f258a6367d204dc64df/e has an old edit so flush to free WALs after random delay 288437 ms 2022-03-05 11:49:48,219 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 821 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 11:51:04,397 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646480463655 with entries=1318, filesize=565.21 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646481064376 2022-03-05 11:51:04,398 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477124574 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477124574 2022-03-05 11:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=324.59 KB, freeSize=306.88 MB, max=307.20 MB, blockCount=11, accesses=7353, hits=6459, hitRatio=87.84%, , cachingAccesses=6492, cachingHits=6442, cachingHitsRatio=99.23%, evictions=2965, evicted=0, evictedPerRun=0.0 2022-03-05 11:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 11:52:02,985 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5. because 7534b3b1812808d2d3c16342fbf347a5/e has an old edit so flush to free WALs after random delay 161017 ms 2022-03-05 11:53:31,414 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 80a1ffe1dc2d8f258a6367d204dc64df 1/1 column families, dataSize=308.62 KB heapSize=576.27 KB 2022-03-05 11:53:31,455 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=308.62 KB at sequenceid=10485 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/.tmp/e/c3b71184707c4f04a0c6169be091302a 2022-03-05 11:53:31,468 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/c3b71184707c4f04a0c6169be091302a, entries=3653, sequenceid=10485, filesize=40.7 K 2022-03-05 11:53:31,469 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~308.62 KB/316031, heapSize ~576.25 KB/590080, currentSize=0 B/0 for 80a1ffe1dc2d8f258a6367d204dc64df in 56ms, sequenceid=10485, compaction requested=true 2022-03-05 11:53:31,470 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 80a1ffe1dc2d8f258a6367d204dc64df/e in aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df. 2022-03-05 11:53:31,470 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/d6a72e9bdc87474492d51122310397fb, hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/5dcde96c4bf7443f997b4b496f352c2c, hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/c3b71184707c4f04a0c6169be091302a] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/.tmp, totalSize=379.5 K 2022-03-05 11:53:31,538 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 80a1ffe1dc2d8f258a6367d204dc64df#e#compaction#80 average throughput is 50.75 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 11:53:31,596 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 80a1ffe1dc2d8f258a6367d204dc64df/e of 80a1ffe1dc2d8f258a6367d204dc64df into deb34122ff904493a89324aaf18b20ad(size=378.0 K), total size for store is 378.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 11:53:31,596 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., storeName=80a1ffe1dc2d8f258a6367d204dc64df/e, priority=13, startTime=1646481211469; duration=0sec 2022-03-05 11:53:42,980 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e. because 4333b489291b1f77671b8f2e78779a1e/__status_family___ has an old edit so flush to free WALs after random delay 25186 ms 2022-03-05 11:53:44,423 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646480623673.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646481224404.meta 2022-03-05 11:53:44,423 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646480623673.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646480623673.meta 2022-03-05 11:54:08,166 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 4333b489291b1f77671b8f2e78779a1e 1/1 column families, dataSize=7.62 KB heapSize=12.16 KB 2022-03-05 11:54:08,215 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=7.62 KB at sequenceid=591 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/.tmp/__status_family___/950da47fb9224c1ba345017b6c205a4f 2022-03-05 11:54:08,247 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/950da47fb9224c1ba345017b6c205a4f, entries=3, sequenceid=591, filesize=5.1 K 2022-03-05 11:54:08,248 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~7.62 KB/7801, heapSize ~12.15 KB/12440, currentSize=0 B/0 for 4333b489291b1f77671b8f2e78779a1e in 82ms, sequenceid=591, compaction requested=false 2022-03-05 11:54:44,002 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 7534b3b1812808d2d3c16342fbf347a5 1/1 column families, dataSize=186.24 KB heapSize=347.02 KB 2022-03-05 11:54:44,075 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=186.24 KB at sequenceid=8491 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/.tmp/e/370855f7ffcd4c1cab1ce3696d5c2209 2022-03-05 11:54:44,087 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/370855f7ffcd4c1cab1ce3696d5c2209, entries=2168, sequenceid=8491, filesize=26.1 K 2022-03-05 11:54:44,088 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~186.24 KB/190713, heapSize ~347 KB/355328, currentSize=0 B/0 for 7534b3b1812808d2d3c16342fbf347a5 in 86ms, sequenceid=8491, compaction requested=false 2022-03-05 11:55:33,406 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. because 7d1b53259f1d8280a47f7704f98f2c9f/e has an old edit so flush to free WALs after random delay 89188 ms 2022-03-05 11:56:33,417 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb. because ccdacf4b414cb8d1bc548efc77c663cb/e has an old edit so flush to free WALs after random delay 80191 ms 2022-03-05 11:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=323.46 KB, freeSize=306.88 MB, max=307.20 MB, blockCount=10, accesses=7420, hits=6481, hitRatio=87.35%, , cachingAccesses=6514, cachingHits=6464, cachingHitsRatio=99.23%, evictions=2995, evicted=0, evictedPerRun=0.0 2022-03-05 11:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 11:57:02,594 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 7d1b53259f1d8280a47f7704f98f2c9f 1/1 column families, dataSize=654.06 KB heapSize=1.10 MB 2022-03-05 11:57:02,672 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=654.06 KB at sequenceid=10061 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/.tmp/e/4425ce9dd3d943b4bb403a8ac69af590 2022-03-05 11:57:02,686 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/4425ce9dd3d943b4bb403a8ac69af590, entries=6300, sequenceid=10061, filesize=63.5 K 2022-03-05 11:57:02,687 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~654.06 KB/669755, heapSize ~1.10 MB/1148328, currentSize=0 B/0 for 7d1b53259f1d8280a47f7704f98f2c9f in 93ms, sequenceid=10061, compaction requested=false 2022-03-05 11:57:53,609 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing ccdacf4b414cb8d1bc548efc77c663cb 1/1 column families, dataSize=296.33 KB heapSize=518.28 KB 2022-03-05 11:57:53,659 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=296.33 KB at sequenceid=5826 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/.tmp/e/0a6d370f8abb466782d56e4c0c2bf7e7 2022-03-05 11:57:53,671 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/0a6d370f8abb466782d56e4c0c2bf7e7, entries=3010, sequenceid=5826, filesize=32.2 K 2022-03-05 11:57:53,672 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~296.33 KB/303446, heapSize ~518.27 KB/530704, currentSize=0 B/0 for ccdacf4b414cb8d1bc548efc77c663cb in 63ms, sequenceid=5826, compaction requested=true 2022-03-05 11:57:53,673 INFO [regionserver/cvp328:16201-longCompactions-0] regionserver.HRegion: Starting compaction of ccdacf4b414cb8d1bc548efc77c663cb/e in aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb. 2022-03-05 11:57:53,673 INFO [regionserver/cvp328:16201-longCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/4927e58e7534492a9d34d845b5af0c95, hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/0455b4138e6048bd9e1337307e723232, hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/0a6d370f8abb466782d56e4c0c2bf7e7] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/.tmp, totalSize=340.6 K 2022-03-05 11:57:53,743 INFO [regionserver/cvp328:16201-longCompactions-0] throttle.PressureAwareThroughputController: ccdacf4b414cb8d1bc548efc77c663cb#e#compaction#85 average throughput is 54.83 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 11:57:53,796 INFO [regionserver/cvp328:16201-longCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in ccdacf4b414cb8d1bc548efc77c663cb/e of ccdacf4b414cb8d1bc548efc77c663cb into e81da0af56254c759292efa6a6c348ab(size=345.6 K), total size for store is 345.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 11:57:53,796 INFO [regionserver/cvp328:16201-longCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., storeName=ccdacf4b414cb8d1bc548efc77c663cb/e, priority=13, startTime=1646481473672; duration=0sec 2022-03-05 12:01:04,447 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646481064376 with entries=1313, filesize=568.89 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646481664436 2022-03-05 12:01:04,447 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477211445 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477211445 2022-03-05 12:01:04,452 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477364340 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477364340 2022-03-05 12:01:04,456 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477401032 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477401032 2022-03-05 12:01:04,459 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477451125 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477451125 2022-03-05 12:01:04,463 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477769048 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477769048 2022-03-05 12:01:04,470 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477790113 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477790113 2022-03-05 12:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=323.46 KB, freeSize=306.88 MB, max=307.20 MB, blockCount=10, accesses=7486, hits=6503, hitRatio=86.87%, , cachingAccesses=6536, cachingHits=6486, cachingHitsRatio=99.24%, evictions=3025, evicted=0, evictedPerRun=0.0 2022-03-05 12:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 12:02:02,993 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391. because 0a764e51258807790269e46c55b99391/e has an old edit so flush to free WALs after random delay 168543 ms 2022-03-05 12:03:44,480 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646481224404.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646481824464.meta 2022-03-05 12:03:44,480 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646481224404.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646481224404.meta 2022-03-05 12:04:51,536 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 0a764e51258807790269e46c55b99391 1/1 column families, dataSize=934.66 KB heapSize=1.64 MB 2022-03-05 12:04:51,786 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=934.66 KB at sequenceid=19732 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/.tmp/e/f7420eee492a4352b72049e9ab615ee3 2022-03-05 12:04:51,803 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/f7420eee492a4352b72049e9ab615ee3, entries=10054, sequenceid=19732, filesize=92.9 K 2022-03-05 12:04:51,804 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~934.66 KB/957087, heapSize ~1.64 MB/1719744, currentSize=0 B/0 for 0a764e51258807790269e46c55b99391 in 268ms, sequenceid=19732, compaction requested=true 2022-03-05 12:04:51,804 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 0a764e51258807790269e46c55b99391/e in aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391. 2022-03-05 12:04:51,804 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/a822dd0efb984cbf96d685d69058f1a1, hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/b6dc1513470848399c60dc92add59fd4, hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/f7420eee492a4352b72049e9ab615ee3] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/.tmp, totalSize=952.5 K 2022-03-05 12:04:51,955 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 0a764e51258807790269e46c55b99391#e#compaction#87 average throughput is 56.49 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 12:04:52,127 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 0a764e51258807790269e46c55b99391/e of 0a764e51258807790269e46c55b99391 into b94dcf334c62474faca7de6d33018c42(size=1007.7 K), total size for store is 1007.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 12:04:52,127 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391., storeName=0a764e51258807790269e46c55b99391/e, priority=13, startTime=1646481891804; duration=0sec 2022-03-05 12:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=7638, hits=6527, hitRatio=85.45%, , cachingAccesses=6558, cachingHits=6508, cachingHitsRatio=99.24%, evictions=3055, evicted=0, evictedPerRun=0.0 2022-03-05 12:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 12:11:04,531 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646481664436 with entries=1297, filesize=565.55 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646482264516 2022-03-05 12:11:04,532 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477846685 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646477846685 2022-03-05 12:11:04,554 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646478452223 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646478452223 2022-03-05 12:11:04,564 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646478624752 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646478624752 2022-03-05 12:11:04,569 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646479224837 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646479224837 2022-03-05 12:11:04,574 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646479256085 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646479256085 2022-03-05 12:11:04,579 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646479863560 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646479863560 2022-03-05 12:11:44,465 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=7660, hits=6549, hitRatio=85.50%, , cachingAccesses=6580, cachingHits=6530, cachingHitsRatio=99.24%, evictions=3085, evicted=0, evictedPerRun=0.0 2022-03-05 12:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 12:13:44,575 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646481824464.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646482424556.meta 2022-03-05 12:13:44,575 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646481824464.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646481824464.meta 2022-03-05 12:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=7682, hits=6571, hitRatio=85.54%, , cachingAccesses=6602, cachingHits=6552, cachingHitsRatio=99.24%, evictions=3115, evicted=0, evictedPerRun=0.0 2022-03-05 12:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 12:19:03,814 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 656 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 12:19:03,814 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 552 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 12:19:03,814 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 544 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 12:19:03,816 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 511 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 12:19:14,919 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3102ms No GCs detected 2022-03-05 12:19:14,927 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3626 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 12:19:14,947 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3245 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 12:19:17,587 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-05 12:19:47,654 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-05 12:21:07,088 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646482264516 with entries=1286, filesize=564.74 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646482864590 2022-03-05 12:21:44,384 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=7704, hits=6593, hitRatio=85.58%, , cachingAccesses=6624, cachingHits=6574, cachingHitsRatio=99.25%, evictions=3145, evicted=0, evictedPerRun=0.0 2022-03-05 12:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 12:22:05,267 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2965 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 12:22:06,699 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1188ms No GCs detected 2022-03-05 12:22:06,701 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1433 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 12:23:47,112 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646482424556.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646483027090.meta 2022-03-05 12:23:47,112 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646482424556.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646482424556.meta 2022-03-05 12:25:42,334 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 641 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 12:25:42,334 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 601 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 12:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=7726, hits=6615, hitRatio=85.62%, , cachingAccesses=6646, cachingHits=6596, cachingHitsRatio=99.25%, evictions=3175, evicted=0, evictedPerRun=0.0 2022-03-05 12:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 12:27:03,425 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 572 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 12:31:07,136 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646482864590 with entries=1300, filesize=571.15 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646483467117 2022-03-05 12:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=7748, hits=6637, hitRatio=85.66%, , cachingAccesses=6668, cachingHits=6618, cachingHitsRatio=99.25%, evictions=3205, evicted=0, evictedPerRun=0.0 2022-03-05 12:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-05 12:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 12:33:47,156 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646483027090.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646483627138.meta 2022-03-05 12:33:47,156 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646483027090.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646483027090.meta 2022-03-05 12:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=7776, hits=6665, hitRatio=85.71%, , cachingAccesses=6696, cachingHits=6646, cachingHitsRatio=99.25%, evictions=3235, evicted=0, evictedPerRun=0.0 2022-03-05 12:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 12:41:07,181 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646483467117 with entries=1302, filesize=570.58 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646484067161 2022-03-05 12:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=7798, hits=6687, hitRatio=85.75%, , cachingAccesses=6718, cachingHits=6668, cachingHitsRatio=99.26%, evictions=3265, evicted=0, evictedPerRun=0.0 2022-03-05 12:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 12:43:47,200 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646483627138.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646484227183.meta 2022-03-05 12:43:47,200 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646483627138.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646483627138.meta 2022-03-05 12:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=7820, hits=6709, hitRatio=85.79%, , cachingAccesses=6740, cachingHits=6690, cachingHitsRatio=99.26%, evictions=3295, evicted=0, evictedPerRun=0.0 2022-03-05 12:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 12:46:52,976 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb. because 9b121c22e18472616debd0996ee2a4cb/e has an old edit so flush to free WALs after random delay 108950 ms 2022-03-05 12:48:41,926 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 9b121c22e18472616debd0996ee2a4cb 1/1 column families, dataSize=562.84 KB heapSize=1.02 MB 2022-03-05 12:48:41,996 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=562.84 KB at sequenceid=16010 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/.tmp/e/758d5af9b7a94a15b99e915750995719 2022-03-05 12:48:42,011 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/758d5af9b7a94a15b99e915750995719, entries=6519, sequenceid=16010, filesize=67.8 K 2022-03-05 12:48:42,012 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~562.84 KB/576347, heapSize ~1.02 MB/1065408, currentSize=0 B/0 for 9b121c22e18472616debd0996ee2a4cb in 86ms, sequenceid=16010, compaction requested=false 2022-03-05 12:51:03,764 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2756 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 12:51:03,764 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2756 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 12:51:03,764 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2749 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 12:51:03,764 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2749 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 12:51:03,764 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2749 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 12:51:03,802 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1392 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 12:51:03,802 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 987 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 12:51:03,802 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 987 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 12:51:07,233 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646484067161 with entries=1298, filesize=564.39 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646484667205 2022-03-05 12:51:07,233 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646480463655 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646480463655 2022-03-05 12:51:37,498 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1237 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 12:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=7842, hits=6731, hitRatio=85.83%, , cachingAccesses=6762, cachingHits=6712, cachingHitsRatio=99.26%, evictions=3325, evicted=0, evictedPerRun=0.0 2022-03-05 12:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 12:53:32,979 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df. because 80a1ffe1dc2d8f258a6367d204dc64df/e has an old edit so flush to free WALs after random delay 240575 ms 2022-03-05 12:53:47,445 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646484227183.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646484827420.meta 2022-03-05 12:53:47,445 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646484227183.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646484227183.meta 2022-03-05 12:53:47,462 WARN [Close-WAL-Writer-80] asyncfs.FanOutOneBlockAsyncDFSOutputHelper: complete file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646484227183.meta not finished, retry = 0 2022-03-05 12:54:42,976 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e. because 4333b489291b1f77671b8f2e78779a1e/__status_family___ has an old edit so flush to free WALs after random delay 215810 ms 2022-03-05 12:54:52,977 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5. because 7534b3b1812808d2d3c16342fbf347a5/e has an old edit so flush to free WALs after random delay 176001 ms 2022-03-05 12:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=7864, hits=6753, hitRatio=85.87%, , cachingAccesses=6784, cachingHits=6734, cachingHitsRatio=99.26%, evictions=3355, evicted=0, evictedPerRun=0.0 2022-03-05 12:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 12:57:02,978 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. because 7d1b53259f1d8280a47f7704f98f2c9f/e has an old edit so flush to free WALs after random delay 270047 ms 2022-03-05 12:57:33,554 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 80a1ffe1dc2d8f258a6367d204dc64df 1/1 column families, dataSize=307.95 KB heapSize=575.73 KB 2022-03-05 12:57:33,661 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=307.95 KB at sequenceid=11706 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/.tmp/e/ef14dc58ade44a7aae0cdcc0da2803af 2022-03-05 12:57:33,767 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/ef14dc58ade44a7aae0cdcc0da2803af, entries=3655, sequenceid=11706, filesize=41.1 K 2022-03-05 12:57:33,769 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~307.95 KB/315339, heapSize ~575.72 KB/589536, currentSize=0 B/0 for 80a1ffe1dc2d8f258a6367d204dc64df in 215ms, sequenceid=11706, compaction requested=false 2022-03-05 12:57:48,979 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 7534b3b1812808d2d3c16342fbf347a5 1/1 column families, dataSize=183.07 KB heapSize=340.49 KB 2022-03-05 12:57:49,015 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=183.07 KB at sequenceid=9474 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/.tmp/e/934a77e42e544bdeb092ebc58ec797b4 2022-03-05 12:57:49,028 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/934a77e42e544bdeb092ebc58ec797b4, entries=2122, sequenceid=9474, filesize=25.5 K 2022-03-05 12:57:49,029 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~183.07 KB/187460, heapSize ~340.48 KB/348648, currentSize=0 B/0 for 7534b3b1812808d2d3c16342fbf347a5 in 50ms, sequenceid=9474, compaction requested=true 2022-03-05 12:57:49,030 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 7534b3b1812808d2d3c16342fbf347a5/e in aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5. 2022-03-05 12:57:49,030 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/503211fb70ab41168efb8399e86362bf, hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/370855f7ffcd4c1cab1ce3696d5c2209, hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/934a77e42e544bdeb092ebc58ec797b4] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/.tmp, totalSize=253.1 K 2022-03-05 12:57:49,091 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 7534b3b1812808d2d3c16342fbf347a5#e#compaction#91 average throughput is 54.88 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 12:57:49,533 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 7534b3b1812808d2d3c16342fbf347a5/e of 7534b3b1812808d2d3c16342fbf347a5 into cca2ce0064bc4fc2adfa4e65f7ed6150(size=247.4 K), total size for store is 247.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 12:57:49,533 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., storeName=7534b3b1812808d2d3c16342fbf347a5/e, priority=13, startTime=1646485069029; duration=0sec 2022-03-05 12:58:03,315 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb. because ccdacf4b414cb8d1bc548efc77c663cb/e has an old edit so flush to free WALs after random delay 214822 ms 2022-03-05 12:58:18,786 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 4333b489291b1f77671b8f2e78779a1e 1/1 column families, dataSize=8.00 KB heapSize=12.75 KB 2022-03-05 12:58:18,825 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=8.00 KB at sequenceid=658 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/.tmp/__status_family___/4e665da01d0d4be086f8ea379f4a18ba 2022-03-05 12:58:18,839 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/4e665da01d0d4be086f8ea379f4a18ba, entries=3, sequenceid=658, filesize=5.1 K 2022-03-05 12:58:18,840 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~8.00 KB/8187, heapSize ~12.73 KB/13040, currentSize=0 B/0 for 4333b489291b1f77671b8f2e78779a1e in 54ms, sequenceid=658, compaction requested=true 2022-03-05 12:58:18,840 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 4333b489291b1f77671b8f2e78779a1e/__status_family___ in __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e. 2022-03-05 12:58:18,841 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/6e6329e0b80945efa437da601d441ae6, hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/950da47fb9224c1ba345017b6c205a4f, hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/4e665da01d0d4be086f8ea379f4a18ba] into tmpdir=hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/.tmp, totalSize=15.4 K 2022-03-05 12:58:18,856 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 4333b489291b1f77671b8f2e78779a1e#__status_family___#compaction#93 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 12:58:18,896 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 4333b489291b1f77671b8f2e78779a1e/__status_family___ of 4333b489291b1f77671b8f2e78779a1e into 9b23660baf4147d89b421833b2d6dfec(size=5.2 K), total size for store is 5.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 12:58:18,896 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=__test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., storeName=4333b489291b1f77671b8f2e78779a1e/__status_family___, priority=13, startTime=1646485098840; duration=0sec 2022-03-05 12:58:31,904 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 774 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 12:58:31,904 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 675 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 12:58:31,904 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 647 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 13:01:07,553 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646484667205 with entries=1311, filesize=576.07 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646485267539 2022-03-05 13:01:33,025 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 7d1b53259f1d8280a47f7704f98f2c9f 1/1 column families, dataSize=681.47 KB heapSize=1.14 MB 2022-03-05 13:01:33,087 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=681.47 KB at sequenceid=11189 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/.tmp/e/0c51a0d7d01d4a3d8455d584fdcef1ea 2022-03-05 13:01:33,098 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/0c51a0d7d01d4a3d8455d584fdcef1ea, entries=6569, sequenceid=11189, filesize=65.1 K 2022-03-05 13:01:33,099 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~681.47 KB/697825, heapSize ~1.14 MB/1196824, currentSize=0 B/0 for 7d1b53259f1d8280a47f7704f98f2c9f in 74ms, sequenceid=11189, compaction requested=true 2022-03-05 13:01:33,099 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 7d1b53259f1d8280a47f7704f98f2c9f/e in aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. 2022-03-05 13:01:33,100 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/d7067341838449ddaf582a633d1a2665, hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/4425ce9dd3d943b4bb403a8ac69af590, hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/0c51a0d7d01d4a3d8455d584fdcef1ea] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/.tmp, totalSize=767.5 K 2022-03-05 13:01:33,227 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 7d1b53259f1d8280a47f7704f98f2c9f#e#compaction#95 average throughput is 57.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 13:01:33,291 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 7d1b53259f1d8280a47f7704f98f2c9f/e of 7d1b53259f1d8280a47f7704f98f2c9f into 640d6366ba564fbaa610904178b681f4(size=803.3 K), total size for store is 803.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 13:01:33,291 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., storeName=7d1b53259f1d8280a47f7704f98f2c9f/e, priority=13, startTime=1646485293099; duration=0sec 2022-03-05 13:01:38,137 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing ccdacf4b414cb8d1bc548efc77c663cb 1/1 column families, dataSize=307.17 KB heapSize=536.16 KB 2022-03-05 13:01:39,253 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 885 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 13:01:44,077 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1354ms No GCs detected 2022-03-05 13:01:44,102 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1877 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 13:01:44,306 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=307.17 KB at sequenceid=6576 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/.tmp/e/1237d966c92843618879a63528e566c4 2022-03-05 13:01:44,372 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/1237d966c92843618879a63528e566c4, entries=3107, sequenceid=6576, filesize=32.8 K 2022-03-05 13:01:44,374 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~307.17 KB/314542, heapSize ~536.15 KB/549016, currentSize=0 B/0 for ccdacf4b414cb8d1bc548efc77c663cb in 6237ms, sequenceid=6576, compaction requested=false 2022-03-05 13:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=8022, hits=6775, hitRatio=84.46%, , cachingAccesses=6806, cachingHits=6756, cachingHitsRatio=99.27%, evictions=3385, evicted=0, evictedPerRun=0.0 2022-03-05 13:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 13:03:04,322 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 712 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 13:03:47,576 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646484827420.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646485427555.meta 2022-03-05 13:03:47,576 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646484827420.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646484827420.meta 2022-03-05 13:04:52,995 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391. because 0a764e51258807790269e46c55b99391/e has an old edit so flush to free WALs after random delay 231480 ms 2022-03-05 13:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=8044, hits=6797, hitRatio=84.50%, , cachingAccesses=6828, cachingHits=6778, cachingHitsRatio=99.27%, evictions=3415, evicted=0, evictedPerRun=0.0 2022-03-05 13:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 13:08:44,475 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 0a764e51258807790269e46c55b99391 1/1 column families, dataSize=942.21 KB heapSize=1.65 MB 2022-03-05 13:08:44,599 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=942.21 KB at sequenceid=22174 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/.tmp/e/09d142504014402890df5621b95ef70b 2022-03-05 13:08:44,615 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/09d142504014402890df5621b95ef70b, entries=10124, sequenceid=22174, filesize=93.2 K 2022-03-05 13:08:44,616 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~942.21 KB/964827, heapSize ~1.65 MB/1732776, currentSize=0 B/0 for 0a764e51258807790269e46c55b99391 in 141ms, sequenceid=22174, compaction requested=false 2022-03-05 13:11:07,596 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646485267539 with entries=1295, filesize=569.26 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646485867580 2022-03-05 13:11:07,597 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646481064376 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646481064376 2022-03-05 13:11:07,602 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646481664436 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646481664436 2022-03-05 13:11:07,606 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646482264516 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646482264516 2022-03-05 13:11:07,612 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646482864590 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646482864590 2022-03-05 13:11:07,618 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646483467117 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646483467117 2022-03-05 13:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=8066, hits=6819, hitRatio=84.54%, , cachingAccesses=6850, cachingHits=6800, cachingHitsRatio=99.27%, evictions=3444, evicted=0, evictedPerRun=0.0 2022-03-05 13:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 13:13:47,620 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646485427555.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646486027600.meta 2022-03-05 13:13:47,620 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646485427555.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646485427555.meta 2022-03-05 13:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=8088, hits=6841, hitRatio=84.58%, , cachingAccesses=6872, cachingHits=6822, cachingHitsRatio=99.27%, evictions=3474, evicted=0, evictedPerRun=0.0 2022-03-05 13:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 13:21:07,638 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646485867580 with entries=1293, filesize=566.19 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646486467624 2022-03-05 13:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=8110, hits=6863, hitRatio=84.62%, , cachingAccesses=6894, cachingHits=6844, cachingHitsRatio=99.27%, evictions=3504, evicted=0, evictedPerRun=0.0 2022-03-05 13:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 13:23:47,670 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646486027600.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646486627653.meta 2022-03-05 13:23:47,670 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646486027600.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646486027600.meta 2022-03-05 13:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=8132, hits=6885, hitRatio=84.67%, , cachingAccesses=6916, cachingHits=6866, cachingHitsRatio=99.28%, evictions=3534, evicted=0, evictedPerRun=0.0 2022-03-05 13:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 13:28:05,152 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 951 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-05 13:31:07,693 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646486467624 with entries=1287, filesize=570.28 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646487067675 2022-03-05 13:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=8154, hits=6907, hitRatio=84.71%, , cachingAccesses=6938, cachingHits=6888, cachingHitsRatio=99.28%, evictions=3564, evicted=0, evictedPerRun=0.0 2022-03-05 13:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-05 13:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 13:33:47,713 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646486627653.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646487227695.meta 2022-03-05 13:33:47,713 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646486627653.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646486627653.meta 2022-03-05 13:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=8182, hits=6935, hitRatio=84.76%, , cachingAccesses=6966, cachingHits=6916, cachingHitsRatio=99.28%, evictions=3594, evicted=0, evictedPerRun=0.0 2022-03-05 13:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 13:41:07,732 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646487067675 with entries=1294, filesize=571.27 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646487667718 2022-03-05 13:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=8204, hits=6957, hitRatio=84.80%, , cachingAccesses=6988, cachingHits=6938, cachingHitsRatio=99.28%, evictions=3624, evicted=0, evictedPerRun=0.0 2022-03-05 13:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 13:43:47,751 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646487227695.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646487827734.meta 2022-03-05 13:43:47,751 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646487227695.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646487227695.meta 2022-03-05 13:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=8226, hits=6979, hitRatio=84.84%, , cachingAccesses=7010, cachingHits=6960, cachingHitsRatio=99.29%, evictions=3654, evicted=0, evictedPerRun=0.0 2022-03-05 13:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 13:48:42,976 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb. because 9b121c22e18472616debd0996ee2a4cb/e has an old edit so flush to free WALs after random delay 76582 ms 2022-03-05 13:49:59,558 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 9b121c22e18472616debd0996ee2a4cb 1/1 column families, dataSize=565.43 KB heapSize=1.02 MB 2022-03-05 13:49:59,617 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=565.43 KB at sequenceid=17648 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/.tmp/e/cbf2d6df37e149938b21335509d7ac4b 2022-03-05 13:49:59,629 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/cbf2d6df37e149938b21335509d7ac4b, entries=6570, sequenceid=17648, filesize=68.0 K 2022-03-05 13:49:59,631 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~565.43 KB/578997, heapSize ~1.02 MB/1071928, currentSize=0 B/0 for 9b121c22e18472616debd0996ee2a4cb in 73ms, sequenceid=17648, compaction requested=true 2022-03-05 13:49:59,631 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 9b121c22e18472616debd0996ee2a4cb/e in aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb. 2022-03-05 13:49:59,631 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/81bc2237797d4256b2edf2d91192e745, hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/758d5af9b7a94a15b99e915750995719, hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/cbf2d6df37e149938b21335509d7ac4b] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/.tmp, totalSize=791.4 K 2022-03-05 13:49:59,752 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 9b121c22e18472616debd0996ee2a4cb#e#compaction#99 average throughput is 55.80 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 13:49:59,828 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 9b121c22e18472616debd0996ee2a4cb/e of 9b121c22e18472616debd0996ee2a4cb into 54ba65620ed1421e8b6f5f390e79eb4f(size=801.0 K), total size for store is 801.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 13:49:59,828 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., storeName=9b121c22e18472616debd0996ee2a4cb/e, priority=13, startTime=1646488199631; duration=0sec 2022-03-05 13:51:07,774 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646487667718 with entries=1298, filesize=562.40 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646488267756 2022-03-05 13:51:07,774 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646484067161 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646484067161 2022-03-05 13:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=8347, hits=7001, hitRatio=83.87%, , cachingAccesses=7032, cachingHits=6982, cachingHitsRatio=99.29%, evictions=3684, evicted=0, evictedPerRun=0.0 2022-03-05 13:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 13:53:47,801 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646487827734.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646488427775.meta 2022-03-05 13:53:47,801 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646487827734.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646487827734.meta 2022-03-05 13:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=8369, hits=7023, hitRatio=83.92%, , cachingAccesses=7054, cachingHits=7004, cachingHitsRatio=99.29%, evictions=3714, evicted=0, evictedPerRun=0.0 2022-03-05 13:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 13:57:53,006 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df. because 80a1ffe1dc2d8f258a6367d204dc64df/e has an old edit so flush to free WALs after random delay 267142 ms 2022-03-05 13:58:02,976 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5. because 7534b3b1812808d2d3c16342fbf347a5/e has an old edit so flush to free WALs after random delay 33048 ms 2022-03-05 13:58:36,024 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 7534b3b1812808d2d3c16342fbf347a5 1/1 column families, dataSize=174.16 KB heapSize=323.58 KB 2022-03-05 13:58:36,187 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=174.16 KB at sequenceid=10425 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/.tmp/e/4b5a271bf5394d009571adf3a55a5ef2 2022-03-05 13:58:36,260 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/4b5a271bf5394d009571adf3a55a5ef2, entries=2013, sequenceid=10425, filesize=24.4 K 2022-03-05 13:58:36,270 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~174.16 KB/178344, heapSize ~323.56 KB/331328, currentSize=0 B/0 for 7534b3b1812808d2d3c16342fbf347a5 in 246ms, sequenceid=10425, compaction requested=false 2022-03-05 13:58:43,017 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e. because 4333b489291b1f77671b8f2e78779a1e/__status_family___ has an old edit so flush to free WALs after random delay 98089 ms 2022-03-05 14:00:21,107 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 4333b489291b1f77671b8f2e78779a1e 1/1 column families, dataSize=7.74 KB heapSize=12.36 KB 2022-03-05 14:00:21,157 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=7.74 KB at sequenceid=724 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/.tmp/__status_family___/b880a1ba218548be901ab7744e880cbf 2022-03-05 14:00:21,166 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/b880a1ba218548be901ab7744e880cbf, entries=3, sequenceid=724, filesize=5.1 K 2022-03-05 14:00:21,168 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~7.74 KB/7929, heapSize ~12.34 KB/12640, currentSize=0 B/0 for 4333b489291b1f77671b8f2e78779a1e in 61ms, sequenceid=724, compaction requested=false 2022-03-05 14:01:07,828 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646488267756 with entries=1287, filesize=568.44 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646488867806 2022-03-05 14:01:43,396 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. because 7d1b53259f1d8280a47f7704f98f2c9f/e has an old edit so flush to free WALs after random delay 47488 ms 2022-03-05 14:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=8391, hits=7045, hitRatio=83.96%, , cachingAccesses=7076, cachingHits=7026, cachingHitsRatio=99.29%, evictions=3744, evicted=0, evictedPerRun=0.0 2022-03-05 14:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 14:01:52,980 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb. because ccdacf4b414cb8d1bc548efc77c663cb/e has an old edit so flush to free WALs after random delay 145996 ms 2022-03-05 14:02:20,148 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 80a1ffe1dc2d8f258a6367d204dc64df 1/1 column families, dataSize=310.74 KB heapSize=579.79 KB 2022-03-05 14:02:20,235 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=310.74 KB at sequenceid=12936 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/.tmp/e/917f20ff326d49d7962edf0990c01db1 2022-03-05 14:02:20,246 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/917f20ff326d49d7962edf0990c01db1, entries=3672, sequenceid=12936, filesize=40.7 K 2022-03-05 14:02:20,247 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~310.74 KB/318198, heapSize ~579.77 KB/593688, currentSize=0 B/0 for 80a1ffe1dc2d8f258a6367d204dc64df in 99ms, sequenceid=12936, compaction requested=true 2022-03-05 14:02:20,248 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 80a1ffe1dc2d8f258a6367d204dc64df/e in aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df. 2022-03-05 14:02:20,248 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/deb34122ff904493a89324aaf18b20ad, hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/ef14dc58ade44a7aae0cdcc0da2803af, hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/917f20ff326d49d7962edf0990c01db1] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/.tmp, totalSize=459.9 K 2022-03-05 14:02:20,323 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 80a1ffe1dc2d8f258a6367d204dc64df#e#compaction#103 average throughput is 57.67 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 14:02:20,384 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 80a1ffe1dc2d8f258a6367d204dc64df/e of 80a1ffe1dc2d8f258a6367d204dc64df into efa2374b66af40798d8b91f8171a6643(size=455.8 K), total size for store is 455.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 14:02:20,384 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., storeName=80a1ffe1dc2d8f258a6367d204dc64df/e, priority=13, startTime=1646488940247; duration=0sec 2022-03-05 14:02:30,884 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 7d1b53259f1d8280a47f7704f98f2c9f 1/1 column families, dataSize=642.72 KB heapSize=1.08 MB 2022-03-05 14:02:30,928 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=642.72 KB at sequenceid=12257 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/.tmp/e/9a646c9a9cd9464fa2ee0b862917bd3b 2022-03-05 14:02:30,937 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/9a646c9a9cd9464fa2ee0b862917bd3b, entries=6183, sequenceid=12257, filesize=61.5 K 2022-03-05 14:02:30,938 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~642.72 KB/658146, heapSize ~1.08 MB/1127848, currentSize=0 B/0 for 7d1b53259f1d8280a47f7704f98f2c9f in 54ms, sequenceid=12257, compaction requested=false 2022-03-05 14:03:47,844 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646488427775.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646489027829.meta 2022-03-05 14:03:47,844 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646488427775.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646488427775.meta 2022-03-05 14:04:18,976 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing ccdacf4b414cb8d1bc548efc77c663cb 1/1 column families, dataSize=295.77 KB heapSize=513.48 KB 2022-03-05 14:04:19,026 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=295.77 KB at sequenceid=7277 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/.tmp/e/1ecfd7d8694147679cd30953cdf16000 2022-03-05 14:04:19,037 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/1ecfd7d8694147679cd30953cdf16000, entries=2955, sequenceid=7277, filesize=31.2 K 2022-03-05 14:04:19,038 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~295.77 KB/302865, heapSize ~513.46 KB/525784, currentSize=0 B/0 for ccdacf4b414cb8d1bc548efc77c663cb in 62ms, sequenceid=7277, compaction requested=true 2022-03-05 14:04:19,039 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of ccdacf4b414cb8d1bc548efc77c663cb/e in aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb. 2022-03-05 14:04:19,039 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/e81da0af56254c759292efa6a6c348ab, hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/1237d966c92843618879a63528e566c4, hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/1ecfd7d8694147679cd30953cdf16000] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/.tmp, totalSize=409.5 K 2022-03-05 14:04:19,106 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: ccdacf4b414cb8d1bc548efc77c663cb#e#compaction#106 average throughput is 61.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 14:04:19,144 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in ccdacf4b414cb8d1bc548efc77c663cb/e of ccdacf4b414cb8d1bc548efc77c663cb into 59e48df54a7941ef8ad30380f6c0199a(size=417.3 K), total size for store is 417.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 14:04:19,145 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., storeName=ccdacf4b414cb8d1bc548efc77c663cb/e, priority=13, startTime=1646489059039; duration=0sec 2022-03-05 14:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=8522, hits=7067, hitRatio=82.93%, , cachingAccesses=7098, cachingHits=7048, cachingHitsRatio=99.30%, evictions=3774, evicted=0, evictedPerRun=0.0 2022-03-05 14:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 14:08:52,988 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391. because 0a764e51258807790269e46c55b99391/e has an old edit so flush to free WALs after random delay 11513 ms 2022-03-05 14:09:04,501 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 0a764e51258807790269e46c55b99391 1/1 column families, dataSize=898.51 KB heapSize=1.58 MB 2022-03-05 14:09:04,558 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=898.51 KB at sequenceid=24481 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/.tmp/e/8e0027f08ae943ed8d020b8201190fc9 2022-03-05 14:09:04,570 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/8e0027f08ae943ed8d020b8201190fc9, entries=9702, sequenceid=24481, filesize=87.1 K 2022-03-05 14:09:04,571 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~898.51 KB/920071, heapSize ~1.58 MB/1655984, currentSize=0 B/0 for 0a764e51258807790269e46c55b99391 in 70ms, sequenceid=24481, compaction requested=true 2022-03-05 14:09:04,571 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 0a764e51258807790269e46c55b99391/e in aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391. 2022-03-05 14:09:04,571 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/b94dcf334c62474faca7de6d33018c42, hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/09d142504014402890df5621b95ef70b, hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/8e0027f08ae943ed8d020b8201190fc9] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/.tmp, totalSize=1.2 M 2022-03-05 14:09:04,756 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 0a764e51258807790269e46c55b99391#e#compaction#108 average throughput is 55.45 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 14:09:04,804 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 0a764e51258807790269e46c55b99391/e of 0a764e51258807790269e46c55b99391 into 71f6a590f692492599b933551378520f(size=1.2 M), total size for store is 1.2 M. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 14:09:04,804 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391., storeName=0a764e51258807790269e46c55b99391/e, priority=13, startTime=1646489344571; duration=0sec 2022-03-05 14:11:07,861 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646488867806 with entries=1296, filesize=561.77 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646489467849 2022-03-05 14:11:07,861 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646484667205 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646484667205 2022-03-05 14:11:07,868 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646485267539 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646485267539 2022-03-05 14:11:07,872 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646485867580 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646485867580 2022-03-05 14:11:07,878 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646486467624 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646486467624 2022-03-05 14:11:07,883 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646487067675 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646487067675 2022-03-05 14:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=8703, hits=7089, hitRatio=81.45%, , cachingAccesses=7120, cachingHits=7070, cachingHitsRatio=99.30%, evictions=3804, evicted=0, evictedPerRun=0.0 2022-03-05 14:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 14:13:47,879 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646489027829.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646489627863.meta 2022-03-05 14:13:47,879 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646489027829.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646489027829.meta 2022-03-05 14:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=8725, hits=7111, hitRatio=81.50%, , cachingAccesses=7142, cachingHits=7092, cachingHitsRatio=99.30%, evictions=3834, evicted=0, evictedPerRun=0.0 2022-03-05 14:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 14:21:07,901 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646489467849 with entries=1277, filesize=559.43 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646490067883 2022-03-05 14:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=8747, hits=7133, hitRatio=81.55%, , cachingAccesses=7164, cachingHits=7114, cachingHitsRatio=99.30%, evictions=3864, evicted=0, evictedPerRun=0.0 2022-03-05 14:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 14:23:47,925 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646489627863.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646490227904.meta 2022-03-05 14:23:47,925 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646489627863.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646489627863.meta 2022-03-05 14:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=8769, hits=7155, hitRatio=81.59%, , cachingAccesses=7186, cachingHits=7136, cachingHitsRatio=99.30%, evictions=3894, evicted=0, evictedPerRun=0.0 2022-03-05 14:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 14:31:07,979 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646490067883 with entries=1266, filesize=562.58 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646490667930 2022-03-05 14:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=8791, hits=7177, hitRatio=81.64%, , cachingAccesses=7208, cachingHits=7158, cachingHitsRatio=99.31%, evictions=3924, evicted=0, evictedPerRun=0.0 2022-03-05 14:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-05 14:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 14:33:47,998 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646490227904.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646490827981.meta 2022-03-05 14:33:47,998 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646490227904.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646490227904.meta 2022-03-05 14:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=8819, hits=7205, hitRatio=81.70%, , cachingAccesses=7236, cachingHits=7186, cachingHitsRatio=99.31%, evictions=3954, evicted=0, evictedPerRun=0.0 2022-03-05 14:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 14:41:08,019 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646490667930 with entries=1285, filesize=569.53 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646491268003 2022-03-05 14:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=8841, hits=7227, hitRatio=81.74%, , cachingAccesses=7258, cachingHits=7208, cachingHitsRatio=99.31%, evictions=3984, evicted=0, evictedPerRun=0.0 2022-03-05 14:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 14:43:48,038 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646490827981.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646491428021.meta 2022-03-05 14:43:48,038 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646490827981.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646490827981.meta 2022-03-05 14:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=8863, hits=7249, hitRatio=81.79%, , cachingAccesses=7280, cachingHits=7230, cachingHitsRatio=99.31%, evictions=4014, evicted=0, evictedPerRun=0.0 2022-03-05 14:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 14:50:02,990 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb. because 9b121c22e18472616debd0996ee2a4cb/e has an old edit so flush to free WALs after random delay 223823 ms 2022-03-05 14:51:08,052 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646491268003 with entries=1288, filesize=566.82 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646491868042 2022-03-05 14:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=8885, hits=7271, hitRatio=81.83%, , cachingAccesses=7302, cachingHits=7252, cachingHitsRatio=99.32%, evictions=4044, evicted=0, evictedPerRun=0.0 2022-03-05 14:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 14:53:46,813 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 9b121c22e18472616debd0996ee2a4cb 1/1 column families, dataSize=584.40 KB heapSize=1.06 MB 2022-03-05 14:53:46,898 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=584.40 KB at sequenceid=19361 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/.tmp/e/7bb87359ca4249a1aefca8996777e7f9 2022-03-05 14:53:46,913 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/7bb87359ca4249a1aefca8996777e7f9, entries=6804, sequenceid=19361, filesize=71.7 K 2022-03-05 14:53:46,914 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~584.40 KB/598421, heapSize ~1.06 MB/1108952, currentSize=0 B/0 for 9b121c22e18472616debd0996ee2a4cb in 101ms, sequenceid=19361, compaction requested=false 2022-03-05 14:53:48,075 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646491428021.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646492028054.meta 2022-03-05 14:53:48,075 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646491428021.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646491428021.meta 2022-03-05 14:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=8907, hits=7293, hitRatio=81.88%, , cachingAccesses=7324, cachingHits=7274, cachingHitsRatio=99.32%, evictions=4074, evicted=0, evictedPerRun=0.0 2022-03-05 14:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 14:58:53,165 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5. because 7534b3b1812808d2d3c16342fbf347a5/e has an old edit so flush to free WALs after random delay 270733 ms 2022-03-05 15:00:42,984 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e. because 4333b489291b1f77671b8f2e78779a1e/__status_family___ has an old edit so flush to free WALs after random delay 229659 ms 2022-03-05 15:01:08,093 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646491868042 with entries=1282, filesize=560.87 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646492468079 2022-03-05 15:01:08,093 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646487667718 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646487667718 2022-03-05 15:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=8929, hits=7315, hitRatio=81.92%, , cachingAccesses=7346, cachingHits=7296, cachingHitsRatio=99.32%, evictions=4104, evicted=0, evictedPerRun=0.0 2022-03-05 15:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 15:02:32,976 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df. because 80a1ffe1dc2d8f258a6367d204dc64df/e has an old edit so flush to free WALs after random delay 182874 ms 2022-03-05 15:02:32,976 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. because 7d1b53259f1d8280a47f7704f98f2c9f/e has an old edit so flush to free WALs after random delay 217473 ms 2022-03-05 15:03:23,898 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 7534b3b1812808d2d3c16342fbf347a5 1/1 column families, dataSize=179.91 KB heapSize=333.05 KB 2022-03-05 15:03:23,944 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=179.91 KB at sequenceid=11438 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/.tmp/e/06444d6a61034d7896f14fe589c5abe8 2022-03-05 15:03:23,955 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/06444d6a61034d7896f14fe589c5abe8, entries=2063, sequenceid=11438, filesize=24.9 K 2022-03-05 15:03:23,956 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~179.91 KB/184230, heapSize ~333.03 KB/341024, currentSize=0 B/0 for 7534b3b1812808d2d3c16342fbf347a5 in 58ms, sequenceid=11438, compaction requested=true 2022-03-05 15:03:23,957 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 7534b3b1812808d2d3c16342fbf347a5/e in aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5. 2022-03-05 15:03:23,957 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/cca2ce0064bc4fc2adfa4e65f7ed6150, hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/4b5a271bf5394d009571adf3a55a5ef2, hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/06444d6a61034d7896f14fe589c5abe8] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/.tmp, totalSize=296.8 K 2022-03-05 15:03:24,012 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 7534b3b1812808d2d3c16342fbf347a5#e#compaction#111 average throughput is 55.30 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 15:03:24,046 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 7534b3b1812808d2d3c16342fbf347a5/e of 7534b3b1812808d2d3c16342fbf347a5 into 0158d5e92cce4aaa83f185c1381f030b(size=289.1 K), total size for store is 289.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 15:03:24,046 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., storeName=7534b3b1812808d2d3c16342fbf347a5/e, priority=13, startTime=1646492603956; duration=0sec 2022-03-05 15:03:48,113 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646492028054.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646492628095.meta 2022-03-05 15:03:48,113 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646492028054.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646492028054.meta 2022-03-05 15:04:32,643 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 4333b489291b1f77671b8f2e78779a1e 1/1 column families, dataSize=7.99 KB heapSize=12.75 KB 2022-03-05 15:04:32,675 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=7.99 KB at sequenceid=791 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/.tmp/__status_family___/8982f4333c3e4749929489a98546ceb9 2022-03-05 15:04:32,685 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/8982f4333c3e4749929489a98546ceb9, entries=3, sequenceid=791, filesize=5.1 K 2022-03-05 15:04:32,687 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~7.99 KB/8184, heapSize ~12.73 KB/13040, currentSize=0 B/0 for 4333b489291b1f77671b8f2e78779a1e in 44ms, sequenceid=791, compaction requested=true 2022-03-05 15:04:32,688 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 4333b489291b1f77671b8f2e78779a1e/__status_family___ in __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e. 2022-03-05 15:04:32,688 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/9b23660baf4147d89b421833b2d6dfec, hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/b880a1ba218548be901ab7744e880cbf, hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/8982f4333c3e4749929489a98546ceb9] into tmpdir=hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/.tmp, totalSize=15.4 K 2022-03-05 15:04:32,697 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 4333b489291b1f77671b8f2e78779a1e#__status_family___#compaction#113 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 15:04:32,727 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 4333b489291b1f77671b8f2e78779a1e/__status_family___ of 4333b489291b1f77671b8f2e78779a1e into 448317cf0563462faa1c0fd94200a8a0(size=5.2 K), total size for store is 5.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 15:04:32,727 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=__test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., storeName=4333b489291b1f77671b8f2e78779a1e/__status_family___, priority=13, startTime=1646492672687; duration=0sec 2022-03-05 15:04:32,978 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb. because ccdacf4b414cb8d1bc548efc77c663cb/e has an old edit so flush to free WALs after random delay 242589 ms 2022-03-05 15:05:35,851 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 80a1ffe1dc2d8f258a6367d204dc64df 1/1 column families, dataSize=302.93 KB heapSize=566.16 KB 2022-03-05 15:05:35,950 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=302.93 KB at sequenceid=14144 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/.tmp/e/dfa20ad713be471b9d443c9dd4321efd 2022-03-05 15:05:35,961 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/dfa20ad713be471b9d443c9dd4321efd, entries=3592, sequenceid=14144, filesize=40.7 K 2022-03-05 15:05:35,994 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~302.93 KB/310204, heapSize ~566.15 KB/579736, currentSize=0 B/0 for 80a1ffe1dc2d8f258a6367d204dc64df in 143ms, sequenceid=14144, compaction requested=false 2022-03-05 15:06:10,450 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 7d1b53259f1d8280a47f7704f98f2c9f 1/1 column families, dataSize=673.30 KB heapSize=1.13 MB 2022-03-05 15:06:10,523 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=673.30 KB at sequenceid=13369 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/.tmp/e/d5a7e1b3c9dc4b7ca123ced159b131d3 2022-03-05 15:06:10,537 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/d5a7e1b3c9dc4b7ca123ced159b131d3, entries=6487, sequenceid=13369, filesize=64.4 K 2022-03-05 15:06:10,538 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~673.30 KB/689464, heapSize ~1.13 MB/1182232, currentSize=0 B/0 for 7d1b53259f1d8280a47f7704f98f2c9f in 88ms, sequenceid=13369, compaction requested=true 2022-03-05 15:06:10,539 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 7d1b53259f1d8280a47f7704f98f2c9f/e in aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. 2022-03-05 15:06:10,539 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/640d6366ba564fbaa610904178b681f4, hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/9a646c9a9cd9464fa2ee0b862917bd3b, hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/d5a7e1b3c9dc4b7ca123ced159b131d3] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/.tmp, totalSize=929.1 K 2022-03-05 15:06:10,675 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 7d1b53259f1d8280a47f7704f98f2c9f#e#compaction#116 average throughput is 60.72 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 15:06:10,717 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 7d1b53259f1d8280a47f7704f98f2c9f/e of 7d1b53259f1d8280a47f7704f98f2c9f into 7bd249235ca24847b88ec12190ee997d(size=965.3 K), total size for store is 965.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 15:06:10,718 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., storeName=7d1b53259f1d8280a47f7704f98f2c9f/e, priority=13, startTime=1646492770538; duration=0sec 2022-03-05 15:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=9113, hits=7337, hitRatio=80.51%, , cachingAccesses=7368, cachingHits=7318, cachingHitsRatio=99.32%, evictions=4134, evicted=0, evictedPerRun=0.0 2022-03-05 15:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 15:08:35,567 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing ccdacf4b414cb8d1bc548efc77c663cb 1/1 column families, dataSize=296.87 KB heapSize=515.16 KB 2022-03-05 15:08:35,823 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=296.87 KB at sequenceid=7944 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/.tmp/e/a83675abe63448b381f42191ae69dd10 2022-03-05 15:08:35,887 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/a83675abe63448b381f42191ae69dd10, entries=2963, sequenceid=7944, filesize=31.1 K 2022-03-05 15:08:35,897 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~296.87 KB/303998, heapSize ~515.14 KB/527504, currentSize=0 B/0 for ccdacf4b414cb8d1bc548efc77c663cb in 330ms, sequenceid=7944, compaction requested=false 2022-03-05 15:09:13,658 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391. because 0a764e51258807790269e46c55b99391/e has an old edit so flush to free WALs after random delay 212779 ms 2022-03-05 15:11:08,132 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646492468079 with entries=1293, filesize=563.72 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646493068118 2022-03-05 15:11:08,133 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646488267756 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646488267756 2022-03-05 15:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=9135, hits=7359, hitRatio=80.56%, , cachingAccesses=7390, cachingHits=7340, cachingHitsRatio=99.32%, evictions=4164, evicted=0, evictedPerRun=0.0 2022-03-05 15:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 15:12:46,437 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 0a764e51258807790269e46c55b99391 1/1 column families, dataSize=938.45 KB heapSize=1.65 MB 2022-03-05 15:12:46,496 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=938.45 KB at sequenceid=26912 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/.tmp/e/d30aeb97495949429f7d304a7c176309 2022-03-05 15:12:46,506 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/d30aeb97495949429f7d304a7c176309, entries=10095, sequenceid=26912, filesize=93.7 K 2022-03-05 15:12:46,507 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~938.45 KB/960975, heapSize ~1.65 MB/1726728, currentSize=0 B/0 for 0a764e51258807790269e46c55b99391 in 70ms, sequenceid=26912, compaction requested=false 2022-03-05 15:13:48,152 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646492628095.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646493228134.meta 2022-03-05 15:13:48,153 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646492628095.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646492628095.meta 2022-03-05 15:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=9157, hits=7381, hitRatio=80.61%, , cachingAccesses=7412, cachingHits=7362, cachingHitsRatio=99.33%, evictions=4194, evicted=0, evictedPerRun=0.0 2022-03-05 15:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 15:21:08,181 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646493068118 with entries=1273, filesize=556.87 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646493668159 2022-03-05 15:21:08,181 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646488867806 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646488867806 2022-03-05 15:21:08,188 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646489467849 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646489467849 2022-03-05 15:21:08,194 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646490067883 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646490067883 2022-03-05 15:21:08,201 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646490667930 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646490667930 2022-03-05 15:21:08,207 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646491268003 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646491268003 2022-03-05 15:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=9179, hits=7403, hitRatio=80.65%, , cachingAccesses=7434, cachingHits=7384, cachingHitsRatio=99.33%, evictions=4224, evicted=0, evictedPerRun=0.0 2022-03-05 15:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 15:23:48,199 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646493228134.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646493828183.meta 2022-03-05 15:23:48,199 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646493228134.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646493228134.meta 2022-03-05 15:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=9201, hits=7425, hitRatio=80.70%, , cachingAccesses=7456, cachingHits=7406, cachingHitsRatio=99.33%, evictions=4254, evicted=0, evictedPerRun=0.0 2022-03-05 15:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 15:31:08,221 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646493668159 with entries=1287, filesize=565.12 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646494268204 2022-03-05 15:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=9223, hits=7447, hitRatio=80.74%, , cachingAccesses=7478, cachingHits=7428, cachingHitsRatio=99.33%, evictions=4284, evicted=0, evictedPerRun=0.0 2022-03-05 15:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-05 15:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 15:33:48,239 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646493828183.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646494428223.meta 2022-03-05 15:33:48,239 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646493828183.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646493828183.meta 2022-03-05 15:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=9251, hits=7475, hitRatio=80.80%, , cachingAccesses=7506, cachingHits=7456, cachingHitsRatio=99.33%, evictions=4314, evicted=0, evictedPerRun=0.0 2022-03-05 15:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 15:41:08,262 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646494268204 with entries=1286, filesize=573.59 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646494868249 2022-03-05 15:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=333.19 KB, freeSize=306.87 MB, max=307.20 MB, blockCount=19, accesses=9361, hits=7573, hitRatio=80.90%, , cachingAccesses=7616, cachingHits=7554, cachingHitsRatio=99.19%, evictions=4344, evicted=0, evictedPerRun=0.0 2022-03-05 15:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 15:43:48,283 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646494428223.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646495028264.meta 2022-03-05 15:43:48,283 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646494428223.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646494428223.meta 2022-03-05 15:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=439.34 KB, freeSize=306.77 MB, max=307.20 MB, blockCount=23, accesses=9411, hits=7619, hitRatio=80.96%, , cachingAccesses=7666, cachingHits=7600, cachingHitsRatio=99.14%, evictions=4374, evicted=0, evictedPerRun=0.0 2022-03-05 15:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 15:51:08,302 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646494868249 with entries=1289, filesize=570.91 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646495468288 2022-03-05 15:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=514.62 KB, freeSize=306.70 MB, max=307.20 MB, blockCount=26, accesses=9466, hits=7671, hitRatio=81.04%, , cachingAccesses=7721, cachingHits=7652, cachingHitsRatio=99.11%, evictions=4404, evicted=0, evictedPerRun=0.0 2022-03-05 15:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 15:53:48,318 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646495028264.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646495628304.meta 2022-03-05 15:53:48,318 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646495028264.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646495028264.meta 2022-03-05 15:53:53,133 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb. because 9b121c22e18472616debd0996ee2a4cb/e has an old edit so flush to free WALs after random delay 299398 ms 2022-03-05 15:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=537.91 KB, freeSize=306.67 MB, max=307.20 MB, blockCount=27, accesses=9518, hits=7722, hitRatio=81.13%, , cachingAccesses=7773, cachingHits=7703, cachingHitsRatio=99.10%, evictions=4434, evicted=0, evictedPerRun=0.0 2022-03-05 15:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 15:58:52,531 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 9b121c22e18472616debd0996ee2a4cb 1/1 column families, dataSize=591.25 KB heapSize=1.07 MB 2022-03-05 15:58:52,595 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=591.25 KB at sequenceid=21097 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/.tmp/e/fb93baffb3e64631ad51105a4077abfa 2022-03-05 15:58:52,605 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/fb93baffb3e64631ad51105a4077abfa, entries=6866, sequenceid=21097, filesize=72.9 K 2022-03-05 15:58:52,606 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~591.25 KB/605441, heapSize ~1.07 MB/1120504, currentSize=0 B/0 for 9b121c22e18472616debd0996ee2a4cb in 75ms, sequenceid=21097, compaction requested=true 2022-03-05 15:58:52,607 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 9b121c22e18472616debd0996ee2a4cb/e in aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb. 2022-03-05 15:58:52,607 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/54ba65620ed1421e8b6f5f390e79eb4f, hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/7bb87359ca4249a1aefca8996777e7f9, hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/fb93baffb3e64631ad51105a4077abfa] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/.tmp, totalSize=945.6 K 2022-03-05 15:58:52,780 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 9b121c22e18472616debd0996ee2a4cb#e#compaction#120 average throughput is 45.50 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 15:58:52,821 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 9b121c22e18472616debd0996ee2a4cb/e of 9b121c22e18472616debd0996ee2a4cb into c58fb63ef4814d8fba29291831e6a53e(size=954.0 K), total size for store is 954.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 15:58:52,821 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., storeName=9b121c22e18472616debd0996ee2a4cb/e, priority=13, startTime=1646495932606; duration=0sec 2022-03-05 16:01:08,339 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646495468288 with entries=1268, filesize=559.70 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646496068323 2022-03-05 16:01:08,339 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646491868042 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646491868042 2022-03-05 16:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=481.55 KB, freeSize=306.73 MB, max=307.20 MB, blockCount=23, accesses=9673, hits=7762, hitRatio=80.24%, , cachingAccesses=7811, cachingHits=7741, cachingHitsRatio=99.10%, evictions=4464, evicted=0, evictedPerRun=0.0 2022-03-05 16:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 16:03:33,071 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5. because 7534b3b1812808d2d3c16342fbf347a5/e has an old edit so flush to free WALs after random delay 100743 ms 2022-03-05 16:03:48,356 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646495628304.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646496228341.meta 2022-03-05 16:03:48,356 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646495628304.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646495628304.meta 2022-03-05 16:04:42,989 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e. because 4333b489291b1f77671b8f2e78779a1e/__status_family___ has an old edit so flush to free WALs after random delay 271374 ms 2022-03-05 16:05:13,814 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 7534b3b1812808d2d3c16342fbf347a5 1/1 column families, dataSize=176.71 KB heapSize=328.38 KB 2022-03-05 16:05:13,865 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=176.71 KB at sequenceid=12408 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/.tmp/e/ce03bf4e8865496ca2b0d830050b85df 2022-03-05 16:05:13,876 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/ce03bf4e8865496ca2b0d830050b85df, entries=2043, sequenceid=12408, filesize=24.7 K 2022-03-05 16:05:13,877 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~176.71 KB/180956, heapSize ~328.36 KB/336240, currentSize=0 B/0 for 7534b3b1812808d2d3c16342fbf347a5 in 63ms, sequenceid=12408, compaction requested=false 2022-03-05 16:05:52,976 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df. because 80a1ffe1dc2d8f258a6367d204dc64df/e has an old edit so flush to free WALs after random delay 273188 ms 2022-03-05 16:06:13,536 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. because 7d1b53259f1d8280a47f7704f98f2c9f/e has an old edit so flush to free WALs after random delay 279655 ms 2022-03-05 16:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=481.55 KB, freeSize=306.73 MB, max=307.20 MB, blockCount=23, accesses=9695, hits=7784, hitRatio=80.29%, , cachingAccesses=7833, cachingHits=7763, cachingHitsRatio=99.11%, evictions=4494, evicted=0, evictedPerRun=0.0 2022-03-05 16:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 16:08:43,491 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb. because ccdacf4b414cb8d1bc548efc77c663cb/e has an old edit so flush to free WALs after random delay 281801 ms 2022-03-05 16:09:14,363 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 4333b489291b1f77671b8f2e78779a1e 1/1 column families, dataSize=8.12 KB heapSize=12.95 KB 2022-03-05 16:09:14,414 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=8.12 KB at sequenceid=860 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/.tmp/__status_family___/9ef295973c714fbcac86a7b3b9a84610 2022-03-05 16:09:14,423 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/9ef295973c714fbcac86a7b3b9a84610, entries=3, sequenceid=860, filesize=5.1 K 2022-03-05 16:09:14,424 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~8.12 KB/8315, heapSize ~12.93 KB/13240, currentSize=0 B/0 for 4333b489291b1f77671b8f2e78779a1e in 61ms, sequenceid=860, compaction requested=false 2022-03-05 16:10:26,165 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 80a1ffe1dc2d8f258a6367d204dc64df 1/1 column families, dataSize=307.61 KB heapSize=574.66 KB 2022-03-05 16:10:26,210 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=307.61 KB at sequenceid=15375 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/.tmp/e/1c4b0999c56a4ddd88f183285fb6b019 2022-03-05 16:10:26,222 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/1c4b0999c56a4ddd88f183285fb6b019, entries=3645, sequenceid=15375, filesize=40.7 K 2022-03-05 16:10:26,223 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~307.61 KB/314996, heapSize ~574.65 KB/588440, currentSize=0 B/0 for 80a1ffe1dc2d8f258a6367d204dc64df in 59ms, sequenceid=15375, compaction requested=true 2022-03-05 16:10:26,224 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 80a1ffe1dc2d8f258a6367d204dc64df/e in aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df. 2022-03-05 16:10:26,224 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/efa2374b66af40798d8b91f8171a6643, hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/dfa20ad713be471b9d443c9dd4321efd, hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/1c4b0999c56a4ddd88f183285fb6b019] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/.tmp, totalSize=537.1 K 2022-03-05 16:10:26,304 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 80a1ffe1dc2d8f258a6367d204dc64df#e#compaction#124 average throughput is 57.96 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 16:10:26,354 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 80a1ffe1dc2d8f258a6367d204dc64df/e of 80a1ffe1dc2d8f258a6367d204dc64df into a619a92efd864c58a4468c8dba1919fa(size=535.1 K), total size for store is 535.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 16:10:26,354 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., storeName=80a1ffe1dc2d8f258a6367d204dc64df/e, priority=13, startTime=1646496626224; duration=0sec 2022-03-05 16:10:53,191 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 7d1b53259f1d8280a47f7704f98f2c9f 1/1 column families, dataSize=680.91 KB heapSize=1.14 MB 2022-03-05 16:10:53,238 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=680.91 KB at sequenceid=14499 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/.tmp/e/1879dbef193e45e5a7248bcbded765ad 2022-03-05 16:10:53,248 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/1879dbef193e45e5a7248bcbded765ad, entries=6567, sequenceid=14499, filesize=65.3 K 2022-03-05 16:10:53,249 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~680.91 KB/697250, heapSize ~1.14 MB/1196104, currentSize=0 B/0 for 7d1b53259f1d8280a47f7704f98f2c9f in 58ms, sequenceid=14499, compaction requested=false 2022-03-05 16:11:08,376 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646496068323 with entries=1293, filesize=560.13 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646496668362 2022-03-05 16:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=481.55 KB, freeSize=306.73 MB, max=307.20 MB, blockCount=23, accesses=9781, hits=7807, hitRatio=79.82%, , cachingAccesses=7855, cachingHits=7785, cachingHitsRatio=99.11%, evictions=4524, evicted=0, evictedPerRun=0.0 2022-03-05 16:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 16:12:52,976 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391. because 0a764e51258807790269e46c55b99391/e has an old edit so flush to free WALs after random delay 189130 ms 2022-03-05 16:13:25,292 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing ccdacf4b414cb8d1bc548efc77c663cb 1/1 column families, dataSize=300.12 KB heapSize=522.10 KB 2022-03-05 16:13:25,341 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=300.12 KB at sequenceid=8598 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/.tmp/e/618f066693064f68b68e1b34ae6c2acc 2022-03-05 16:13:25,352 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/618f066693064f68b68e1b34ae6c2acc, entries=3012, sequenceid=8598, filesize=31.4 K 2022-03-05 16:13:25,354 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~300.12 KB/307319, heapSize ~522.09 KB/534616, currentSize=0 B/0 for ccdacf4b414cb8d1bc548efc77c663cb in 62ms, sequenceid=8598, compaction requested=true 2022-03-05 16:13:25,354 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of ccdacf4b414cb8d1bc548efc77c663cb/e in aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb. 2022-03-05 16:13:25,354 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/59e48df54a7941ef8ad30380f6c0199a, hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/a83675abe63448b381f42191ae69dd10, hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/618f066693064f68b68e1b34ae6c2acc] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/.tmp, totalSize=479.8 K 2022-03-05 16:13:25,430 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: ccdacf4b414cb8d1bc548efc77c663cb#e#compaction#127 average throughput is 60.37 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 16:13:25,469 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in ccdacf4b414cb8d1bc548efc77c663cb/e of ccdacf4b414cb8d1bc548efc77c663cb into 48f4f75e5a23406097753ed12acd0c0f(size=490.3 K), total size for store is 490.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 16:13:25,469 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., storeName=ccdacf4b414cb8d1bc548efc77c663cb/e, priority=13, startTime=1646496805354; duration=0sec 2022-03-05 16:13:48,390 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646496228341.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646496828378.meta 2022-03-05 16:13:48,390 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646496228341.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646496228341.meta 2022-03-05 16:16:02,106 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 0a764e51258807790269e46c55b99391 1/1 column families, dataSize=930.29 KB heapSize=1.63 MB 2022-03-05 16:16:02,166 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=930.29 KB at sequenceid=29328 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/.tmp/e/9c8b0de503fb44e8b6e781e85d63467a 2022-03-05 16:16:02,177 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/9c8b0de503fb44e8b6e781e85d63467a, entries=10011, sequenceid=29328, filesize=92.6 K 2022-03-05 16:16:02,180 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~930.29 KB/952614, heapSize ~1.63 MB/1712080, currentSize=217 B/217 for 0a764e51258807790269e46c55b99391 in 74ms, sequenceid=29328, compaction requested=true 2022-03-05 16:16:02,180 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 0a764e51258807790269e46c55b99391/e in aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391. 2022-03-05 16:16:02,180 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/71f6a590f692492599b933551378520f, hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/d30aeb97495949429f7d304a7c176309, hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/9c8b0de503fb44e8b6e781e85d63467a] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/.tmp, totalSize=1.4 M 2022-03-05 16:16:02,392 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 0a764e51258807790269e46c55b99391#e#compaction#129 average throughput is 57.38 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 16:16:02,451 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 0a764e51258807790269e46c55b99391/e of 0a764e51258807790269e46c55b99391 into 4a9df5f62cc9402fa1608c87fb0e215c(size=1.4 M), total size for store is 1.4 M. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 16:16:02,451 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391., storeName=0a764e51258807790269e46c55b99391/e, priority=13, startTime=1646496962180; duration=0sec 2022-03-05 16:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=453.56 KB, freeSize=306.76 MB, max=307.20 MB, blockCount=18, accesses=10055, hits=7832, hitRatio=77.89%, , cachingAccesses=7877, cachingHits=7807, cachingHitsRatio=99.11%, evictions=4554, evicted=0, evictedPerRun=0.0 2022-03-05 16:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 16:21:08,407 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646496668362 with entries=1291, filesize=563.78 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646497268394 2022-03-05 16:21:08,407 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646492468079 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646492468079 2022-03-05 16:21:08,429 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646493068118 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646493068118 2022-03-05 16:21:08,434 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646493668159 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646493668159 2022-03-05 16:21:08,441 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646494268204 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646494268204 2022-03-05 16:21:08,446 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646494868249 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646494868249 2022-03-05 16:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=389.15 KB, freeSize=306.82 MB, max=307.20 MB, blockCount=13, accesses=10077, hits=7854, hitRatio=77.94%, , cachingAccesses=7899, cachingHits=7829, cachingHitsRatio=99.11%, evictions=4584, evicted=0, evictedPerRun=0.0 2022-03-05 16:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 16:23:48,429 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646496828378.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646497428409.meta 2022-03-05 16:23:48,429 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646496828378.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646496828378.meta 2022-03-05 16:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=389.15 KB, freeSize=306.82 MB, max=307.20 MB, blockCount=13, accesses=10099, hits=7876, hitRatio=77.99%, , cachingAccesses=7921, cachingHits=7851, cachingHitsRatio=99.12%, evictions=4614, evicted=0, evictedPerRun=0.0 2022-03-05 16:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 16:31:08,446 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646497268394 with entries=1291, filesize=564.40 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646497868434 2022-03-05 16:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=389.15 KB, freeSize=306.82 MB, max=307.20 MB, blockCount=13, accesses=10121, hits=7898, hitRatio=78.04%, , cachingAccesses=7943, cachingHits=7873, cachingHitsRatio=99.12%, evictions=4644, evicted=0, evictedPerRun=0.0 2022-03-05 16:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-05 16:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 16:33:48,465 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646497428409.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646498028447.meta 2022-03-05 16:33:48,465 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646497428409.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646497428409.meta 2022-03-05 16:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=389.15 KB, freeSize=306.82 MB, max=307.20 MB, blockCount=13, accesses=10149, hits=7926, hitRatio=78.10%, , cachingAccesses=7971, cachingHits=7901, cachingHitsRatio=99.12%, evictions=4674, evicted=0, evictedPerRun=0.0 2022-03-05 16:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 16:41:08,484 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646497868434 with entries=1274, filesize=570.24 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646498468471 2022-03-05 16:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=389.15 KB, freeSize=306.82 MB, max=307.20 MB, blockCount=13, accesses=10171, hits=7948, hitRatio=78.14%, , cachingAccesses=7993, cachingHits=7923, cachingHitsRatio=99.12%, evictions=4704, evicted=0, evictedPerRun=0.0 2022-03-05 16:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 16:43:48,504 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646498028447.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646498628489.meta 2022-03-05 16:43:48,504 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646498028447.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646498028447.meta 2022-03-05 16:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=389.15 KB, freeSize=306.82 MB, max=307.20 MB, blockCount=13, accesses=10193, hits=7970, hitRatio=78.19%, , cachingAccesses=8015, cachingHits=7945, cachingHitsRatio=99.13%, evictions=4734, evicted=0, evictedPerRun=0.0 2022-03-05 16:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 16:51:08,524 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646498468471 with entries=1287, filesize=561.52 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646499068511 2022-03-05 16:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=389.15 KB, freeSize=306.82 MB, max=307.20 MB, blockCount=13, accesses=10215, hits=7992, hitRatio=78.24%, , cachingAccesses=8037, cachingHits=7967, cachingHitsRatio=99.13%, evictions=4764, evicted=0, evictedPerRun=0.0 2022-03-05 16:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 16:53:48,543 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646498628489.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646499228526.meta 2022-03-05 16:53:48,543 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646498628489.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646498628489.meta 2022-03-05 16:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=389.15 KB, freeSize=306.82 MB, max=307.20 MB, blockCount=13, accesses=10237, hits=8014, hitRatio=78.28%, , cachingAccesses=8059, cachingHits=7989, cachingHitsRatio=99.13%, evictions=4794, evicted=0, evictedPerRun=0.0 2022-03-05 16:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 16:58:53,364 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb. because 9b121c22e18472616debd0996ee2a4cb/e has an old edit so flush to free WALs after random delay 157197 ms 2022-03-05 17:01:08,573 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646499068511 with entries=1290, filesize=567.94 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646499668550 2022-03-05 17:01:30,562 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 9b121c22e18472616debd0996ee2a4cb 1/1 column families, dataSize=573.78 KB heapSize=1.04 MB 2022-03-05 17:01:30,626 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=573.78 KB at sequenceid=22780 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/.tmp/e/0e64e30905ac48be8b9cbbcefc0ded24 2022-03-05 17:01:30,636 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/0e64e30905ac48be8b9cbbcefc0ded24, entries=6676, sequenceid=22780, filesize=68.7 K 2022-03-05 17:01:30,638 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~573.78 KB/587549, heapSize ~1.04 MB/1088576, currentSize=0 B/0 for 9b121c22e18472616debd0996ee2a4cb in 76ms, sequenceid=22780, compaction requested=false 2022-03-05 17:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=389.15 KB, freeSize=306.82 MB, max=307.20 MB, blockCount=13, accesses=10259, hits=8036, hitRatio=78.33%, , cachingAccesses=8081, cachingHits=8011, cachingHitsRatio=99.13%, evictions=4824, evicted=0, evictedPerRun=0.0 2022-03-05 17:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 17:03:48,596 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646499228526.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646499828575.meta 2022-03-05 17:03:48,596 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646499228526.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646499228526.meta 2022-03-05 17:05:23,203 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5. because 7534b3b1812808d2d3c16342fbf347a5/e has an old edit so flush to free WALs after random delay 297323 ms 2022-03-05 17:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=389.15 KB, freeSize=306.82 MB, max=307.20 MB, blockCount=13, accesses=10281, hits=8058, hitRatio=78.38%, , cachingAccesses=8103, cachingHits=8033, cachingHitsRatio=99.14%, evictions=4854, evicted=0, evictedPerRun=0.0 2022-03-05 17:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 17:09:43,418 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e. because 4333b489291b1f77671b8f2e78779a1e/__status_family___ has an old edit so flush to free WALs after random delay 241444 ms 2022-03-05 17:10:20,526 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 7534b3b1812808d2d3c16342fbf347a5 1/1 column families, dataSize=178.83 KB heapSize=330.70 KB 2022-03-05 17:10:20,647 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=178.83 KB at sequenceid=13423 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/.tmp/e/0272b71b68894462ab742c5aa0bec2b0 2022-03-05 17:10:20,664 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/0272b71b68894462ab742c5aa0bec2b0, entries=2045, sequenceid=13423, filesize=24.7 K 2022-03-05 17:10:20,665 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~178.83 KB/183126, heapSize ~330.69 KB/338624, currentSize=0 B/0 for 7534b3b1812808d2d3c16342fbf347a5 in 139ms, sequenceid=13423, compaction requested=true 2022-03-05 17:10:20,665 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 7534b3b1812808d2d3c16342fbf347a5/e in aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5. 2022-03-05 17:10:20,665 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/0158d5e92cce4aaa83f185c1381f030b, hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/ce03bf4e8865496ca2b0d830050b85df, hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/0272b71b68894462ab742c5aa0bec2b0] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/.tmp, totalSize=338.5 K 2022-03-05 17:10:20,721 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 7534b3b1812808d2d3c16342fbf347a5#e#compaction#132 average throughput is 59.62 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 17:10:21,020 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 7534b3b1812808d2d3c16342fbf347a5/e of 7534b3b1812808d2d3c16342fbf347a5 into 02617b84f0824268bcca4affb7792c63(size=331.3 K), total size for store is 331.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 17:10:21,020 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., storeName=7534b3b1812808d2d3c16342fbf347a5/e, priority=13, startTime=1646500220665; duration=0sec 2022-03-05 17:10:32,978 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df. because 80a1ffe1dc2d8f258a6367d204dc64df/e has an old edit so flush to free WALs after random delay 52192 ms 2022-03-05 17:11:02,976 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. because 7d1b53259f1d8280a47f7704f98f2c9f/e has an old edit so flush to free WALs after random delay 250038 ms 2022-03-05 17:11:08,617 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646499668550 with entries=1288, filesize=560.97 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646500268603 2022-03-05 17:11:08,617 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646495468288 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646495468288 2022-03-05 17:11:25,174 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 80a1ffe1dc2d8f258a6367d204dc64df 1/1 column families, dataSize=291.87 KB heapSize=545.20 KB 2022-03-05 17:11:25,217 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=291.87 KB at sequenceid=16548 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/.tmp/e/68cc4adf77e84abd8089fc548616e2da 2022-03-05 17:11:25,226 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/68cc4adf77e84abd8089fc548616e2da, entries=3457, sequenceid=16548, filesize=38.3 K 2022-03-05 17:11:25,227 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~291.87 KB/298873, heapSize ~545.19 KB/558272, currentSize=0 B/0 for 80a1ffe1dc2d8f258a6367d204dc64df in 53ms, sequenceid=16548, compaction requested=false 2022-03-05 17:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=389.15 KB, freeSize=306.82 MB, max=307.20 MB, blockCount=13, accesses=10345, hits=8082, hitRatio=78.12%, , cachingAccesses=8125, cachingHits=8055, cachingHitsRatio=99.14%, evictions=4884, evicted=0, evictedPerRun=0.0 2022-03-05 17:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 17:13:33,414 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb. because ccdacf4b414cb8d1bc548efc77c663cb/e has an old edit so flush to free WALs after random delay 198701 ms 2022-03-05 17:13:44,863 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 4333b489291b1f77671b8f2e78779a1e 1/1 column families, dataSize=8.12 KB heapSize=12.95 KB 2022-03-05 17:13:44,910 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=8.12 KB at sequenceid=928 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/.tmp/__status_family___/fb2f63a7d46e4790a71a8c6f12d8846d 2022-03-05 17:13:44,919 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/fb2f63a7d46e4790a71a8c6f12d8846d, entries=3, sequenceid=928, filesize=5.1 K 2022-03-05 17:13:44,919 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~8.12 KB/8313, heapSize ~12.93 KB/13240, currentSize=0 B/0 for 4333b489291b1f77671b8f2e78779a1e in 57ms, sequenceid=928, compaction requested=true 2022-03-05 17:13:44,920 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 4333b489291b1f77671b8f2e78779a1e/__status_family___ in __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e. 2022-03-05 17:13:44,920 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/448317cf0563462faa1c0fd94200a8a0, hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/9ef295973c714fbcac86a7b3b9a84610, hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/fb2f63a7d46e4790a71a8c6f12d8846d] into tmpdir=hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/.tmp, totalSize=15.4 K 2022-03-05 17:13:44,930 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 4333b489291b1f77671b8f2e78779a1e#__status_family___#compaction#135 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 17:13:44,967 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 4333b489291b1f77671b8f2e78779a1e/__status_family___ of 4333b489291b1f77671b8f2e78779a1e into 50f254634736454dbf0ade62b5b45b13(size=5.2 K), total size for store is 5.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 17:13:44,967 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=__test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., storeName=4333b489291b1f77671b8f2e78779a1e/__status_family___, priority=13, startTime=1646500424920; duration=0sec 2022-03-05 17:13:48,628 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646499828575.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646500428619.meta 2022-03-05 17:13:48,628 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646499828575.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646499828575.meta 2022-03-05 17:15:13,015 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 7d1b53259f1d8280a47f7704f98f2c9f 1/1 column families, dataSize=683.49 KB heapSize=1.15 MB 2022-03-05 17:15:13,061 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=683.49 KB at sequenceid=15633 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/.tmp/e/1620adedd3d948cfa4d288b2b03b828b 2022-03-05 17:15:13,070 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/1620adedd3d948cfa4d288b2b03b828b, entries=6598, sequenceid=15633, filesize=65.5 K 2022-03-05 17:15:13,071 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~683.49 KB/699893, heapSize ~1.15 MB/1201080, currentSize=0 B/0 for 7d1b53259f1d8280a47f7704f98f2c9f in 56ms, sequenceid=15633, compaction requested=true 2022-03-05 17:15:13,071 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 7d1b53259f1d8280a47f7704f98f2c9f/e in aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. 2022-03-05 17:15:13,071 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/7bd249235ca24847b88ec12190ee997d, hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/1879dbef193e45e5a7248bcbded765ad, hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/1620adedd3d948cfa4d288b2b03b828b] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/.tmp, totalSize=1.1 M 2022-03-05 17:15:13,224 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 7d1b53259f1d8280a47f7704f98f2c9f#e#compaction#137 average throughput is 62.85 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 17:15:13,289 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 7d1b53259f1d8280a47f7704f98f2c9f/e of 7d1b53259f1d8280a47f7704f98f2c9f into d5b6ae6ab3574842bd7776023a5ab09c(size=1.1 M), total size for store is 1.1 M. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 17:15:13,289 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., storeName=7d1b53259f1d8280a47f7704f98f2c9f/e, priority=13, startTime=1646500513071; duration=0sec 2022-03-05 17:16:03,343 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391. because 0a764e51258807790269e46c55b99391/e has an old edit so flush to free WALs after random delay 187956 ms 2022-03-05 17:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=10514, hits=8106, hitRatio=77.10%, , cachingAccesses=8147, cachingHits=8077, cachingHitsRatio=99.14%, evictions=4914, evicted=0, evictedPerRun=0.0 2022-03-05 17:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 17:16:52,116 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing ccdacf4b414cb8d1bc548efc77c663cb 1/1 column families, dataSize=294.67 KB heapSize=510.18 KB 2022-03-05 17:16:52,160 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=294.67 KB at sequenceid=9251 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/.tmp/e/2f9e5226cb5e4f9594530b77b11591ec 2022-03-05 17:16:52,170 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/2f9e5226cb5e4f9594530b77b11591ec, entries=2925, sequenceid=9251, filesize=30.7 K 2022-03-05 17:16:52,171 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~294.67 KB/301742, heapSize ~510.16 KB/522408, currentSize=0 B/0 for ccdacf4b414cb8d1bc548efc77c663cb in 55ms, sequenceid=9251, compaction requested=false 2022-03-05 17:19:11,299 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 0a764e51258807790269e46c55b99391 1/1 column families, dataSize=930.09 KB heapSize=1.63 MB 2022-03-05 17:19:11,366 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=930.09 KB at sequenceid=31744 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/.tmp/e/ce76b136df3a40a4979c15a721b5ea1c 2022-03-05 17:19:11,376 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/ce76b136df3a40a4979c15a721b5ea1c, entries=10009, sequenceid=31744, filesize=92.7 K 2022-03-05 17:19:11,377 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~930.09 KB/952408, heapSize ~1.63 MB/1711656, currentSize=0 B/0 for 0a764e51258807790269e46c55b99391 in 78ms, sequenceid=31744, compaction requested=false 2022-03-05 17:21:08,648 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646500268603 with entries=1280, filesize=555.13 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646500868635 2022-03-05 17:21:08,649 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646496068323 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646496068323 2022-03-05 17:21:08,658 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646496668362 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646496668362 2022-03-05 17:21:08,677 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646497268394 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646497268394 2022-03-05 17:21:08,689 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646497868434 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646497868434 2022-03-05 17:21:08,697 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646498468471 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646498468471 2022-03-05 17:21:08,711 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646499068511 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646499068511 2022-03-05 17:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=10536, hits=8128, hitRatio=77.15%, , cachingAccesses=8169, cachingHits=8099, cachingHitsRatio=99.14%, evictions=4944, evicted=0, evictedPerRun=0.0 2022-03-05 17:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 17:23:48,668 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646500428619.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646501028650.meta 2022-03-05 17:23:48,668 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646500428619.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646500428619.meta 2022-03-05 17:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=10558, hits=8150, hitRatio=77.19%, , cachingAccesses=8191, cachingHits=8121, cachingHitsRatio=99.15%, evictions=4974, evicted=0, evictedPerRun=0.0 2022-03-05 17:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 17:31:08,691 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646500868635 with entries=1292, filesize=567.11 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646501468673 2022-03-05 17:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=10580, hits=8172, hitRatio=77.24%, , cachingAccesses=8213, cachingHits=8143, cachingHitsRatio=99.15%, evictions=5004, evicted=0, evictedPerRun=0.0 2022-03-05 17:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-05 17:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 17:33:48,715 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646501028650.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646501628694.meta 2022-03-05 17:33:48,715 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646501028650.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646501028650.meta 2022-03-05 17:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=10608, hits=8200, hitRatio=77.30%, , cachingAccesses=8241, cachingHits=8171, cachingHitsRatio=99.15%, evictions=5034, evicted=0, evictedPerRun=0.0 2022-03-05 17:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 17:41:08,734 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646501468673 with entries=1275, filesize=570 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646502068721 2022-03-05 17:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=10630, hits=8222, hitRatio=77.35%, , cachingAccesses=8263, cachingHits=8193, cachingHitsRatio=99.15%, evictions=5064, evicted=0, evictedPerRun=0.0 2022-03-05 17:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 17:43:48,757 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646501628694.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646502228736.meta 2022-03-05 17:43:48,757 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646501628694.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646501628694.meta 2022-03-05 17:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=10652, hits=8244, hitRatio=77.39%, , cachingAccesses=8285, cachingHits=8215, cachingHitsRatio=99.16%, evictions=5094, evicted=0, evictedPerRun=0.0 2022-03-05 17:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 17:51:08,782 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646502068721 with entries=1293, filesize=570.23 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646502668762 2022-03-05 17:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=10674, hits=8266, hitRatio=77.44%, , cachingAccesses=8307, cachingHits=8237, cachingHitsRatio=99.16%, evictions=5124, evicted=0, evictedPerRun=0.0 2022-03-05 17:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 17:53:05,547 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2381 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 17:53:05,547 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1431 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 17:53:48,804 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646502228736.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646502828784.meta 2022-03-05 17:53:48,804 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646502228736.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646502228736.meta 2022-03-05 17:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=10696, hits=8288, hitRatio=77.49%, , cachingAccesses=8329, cachingHits=8259, cachingHitsRatio=99.16%, evictions=5154, evicted=0, evictedPerRun=0.0 2022-03-05 17:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 18:01:08,824 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646502668762 with entries=1278, filesize=571.00 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646503268812 2022-03-05 18:01:33,350 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb. because 9b121c22e18472616debd0996ee2a4cb/e has an old edit so flush to free WALs after random delay 156984 ms 2022-03-05 18:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=10718, hits=8310, hitRatio=77.53%, , cachingAccesses=8351, cachingHits=8281, cachingHitsRatio=99.16%, evictions=5184, evicted=0, evictedPerRun=0.0 2022-03-05 18:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 18:03:48,841 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646502828784.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646503428828.meta 2022-03-05 18:03:48,841 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646502828784.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646502828784.meta 2022-03-05 18:04:10,334 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 9b121c22e18472616debd0996ee2a4cb 1/1 column families, dataSize=567.72 KB heapSize=1.03 MB 2022-03-05 18:04:10,409 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=567.72 KB at sequenceid=24451 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/.tmp/e/30c4f77b72a2484d84b6e3755695809c 2022-03-05 18:04:10,423 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/30c4f77b72a2484d84b6e3755695809c, entries=6590, sequenceid=24451, filesize=68.6 K 2022-03-05 18:04:10,424 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~567.72 KB/581343, heapSize ~1.03 MB/1075736, currentSize=0 B/0 for 9b121c22e18472616debd0996ee2a4cb in 90ms, sequenceid=24451, compaction requested=true 2022-03-05 18:04:10,425 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 9b121c22e18472616debd0996ee2a4cb/e in aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb. 2022-03-05 18:04:10,425 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/c58fb63ef4814d8fba29291831e6a53e, hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/0e64e30905ac48be8b9cbbcefc0ded24, hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/30c4f77b72a2484d84b6e3755695809c] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/.tmp, totalSize=1.1 M 2022-03-05 18:04:10,589 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 9b121c22e18472616debd0996ee2a4cb#e#compaction#141 average throughput is 54.14 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 18:04:10,655 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 9b121c22e18472616debd0996ee2a4cb/e of 9b121c22e18472616debd0996ee2a4cb into b9f34279856044949a4426bd55b16f0e(size=1.1 M), total size for store is 1.1 M. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 18:04:10,655 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., storeName=9b121c22e18472616debd0996ee2a4cb/e, priority=13, startTime=1646503450424; duration=0sec 2022-03-05 18:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=10876, hits=8332, hitRatio=76.61%, , cachingAccesses=8373, cachingHits=8303, cachingHitsRatio=99.16%, evictions=5214, evicted=0, evictedPerRun=0.0 2022-03-05 18:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 18:10:32,999 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5. because 7534b3b1812808d2d3c16342fbf347a5/e has an old edit so flush to free WALs after random delay 12646 ms 2022-03-05 18:10:45,645 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 7534b3b1812808d2d3c16342fbf347a5 1/1 column families, dataSize=177.60 KB heapSize=331.16 KB 2022-03-05 18:10:45,704 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=177.60 KB at sequenceid=14389 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/.tmp/e/fb105568e4b74fae81af58505f164382 2022-03-05 18:10:45,724 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/fb105568e4b74fae81af58505f164382, entries=2068, sequenceid=14389, filesize=25.0 K 2022-03-05 18:10:45,730 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~177.60 KB/181862, heapSize ~331.14 KB/339088, currentSize=0 B/0 for 7534b3b1812808d2d3c16342fbf347a5 in 85ms, sequenceid=14389, compaction requested=false 2022-03-05 18:11:08,864 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646503268812 with entries=1297, filesize=560.19 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646503868852 2022-03-05 18:11:08,865 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646499668550 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646499668550 2022-03-05 18:11:33,104 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df. because 80a1ffe1dc2d8f258a6367d204dc64df/e has an old edit so flush to free WALs after random delay 58207 ms 2022-03-05 18:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=10898, hits=8354, hitRatio=76.66%, , cachingAccesses=8395, cachingHits=8325, cachingHitsRatio=99.17%, evictions=5244, evicted=0, evictedPerRun=0.0 2022-03-05 18:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 18:12:31,311 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 80a1ffe1dc2d8f258a6367d204dc64df 1/1 column families, dataSize=295.50 KB heapSize=552.61 KB 2022-03-05 18:12:31,346 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=295.50 KB at sequenceid=17716 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/.tmp/e/c08bcd81b85a4c1b802d5b6d5aef9e5d 2022-03-05 18:12:31,357 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/c08bcd81b85a4c1b802d5b6d5aef9e5d, entries=3509, sequenceid=17716, filesize=38.9 K 2022-03-05 18:12:31,358 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~295.50 KB/302594, heapSize ~552.59 KB/565856, currentSize=0 B/0 for 80a1ffe1dc2d8f258a6367d204dc64df in 47ms, sequenceid=17716, compaction requested=true 2022-03-05 18:12:31,358 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 80a1ffe1dc2d8f258a6367d204dc64df/e in aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df. 2022-03-05 18:12:31,358 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/a619a92efd864c58a4468c8dba1919fa, hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/68cc4adf77e84abd8089fc548616e2da, hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/c08bcd81b85a4c1b802d5b6d5aef9e5d] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/.tmp, totalSize=612.4 K 2022-03-05 18:12:31,455 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 80a1ffe1dc2d8f258a6367d204dc64df#e#compaction#144 average throughput is 55.06 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 18:12:31,580 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 80a1ffe1dc2d8f258a6367d204dc64df/e of 80a1ffe1dc2d8f258a6367d204dc64df into 30b8b29a6df0406d8538da90cf648405(size=608.1 K), total size for store is 608.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 18:12:31,580 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., storeName=80a1ffe1dc2d8f258a6367d204dc64df/e, priority=13, startTime=1646503951358; duration=0sec 2022-03-05 18:13:48,882 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646503428828.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646504028869.meta 2022-03-05 18:13:48,882 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646503428828.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646503428828.meta 2022-03-05 18:14:43,217 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e. because 4333b489291b1f77671b8f2e78779a1e/__status_family___ has an old edit so flush to free WALs after random delay 135012 ms 2022-03-05 18:15:23,516 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. because 7d1b53259f1d8280a47f7704f98f2c9f/e has an old edit so flush to free WALs after random delay 238807 ms 2022-03-05 18:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=10995, hits=8376, hitRatio=76.18%, , cachingAccesses=8417, cachingHits=8347, cachingHitsRatio=99.17%, evictions=5274, evicted=0, evictedPerRun=0.0 2022-03-05 18:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 18:16:58,228 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 4333b489291b1f77671b8f2e78779a1e 1/1 column families, dataSize=7.87 KB heapSize=12.55 KB 2022-03-05 18:16:58,280 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=7.87 KB at sequenceid=995 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/.tmp/__status_family___/e88261b886a346ada815ba08ffb7c345 2022-03-05 18:16:58,290 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/e88261b886a346ada815ba08ffb7c345, entries=3, sequenceid=995, filesize=5.1 K 2022-03-05 18:16:58,291 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~7.87 KB/8057, heapSize ~12.54 KB/12840, currentSize=0 B/0 for 4333b489291b1f77671b8f2e78779a1e in 63ms, sequenceid=995, compaction requested=false 2022-03-05 18:17:03,526 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb. because ccdacf4b414cb8d1bc548efc77c663cb/e has an old edit so flush to free WALs after random delay 137634 ms 2022-03-05 18:19:13,368 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391. because 0a764e51258807790269e46c55b99391/e has an old edit so flush to free WALs after random delay 268806 ms 2022-03-05 18:19:21,160 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing ccdacf4b414cb8d1bc548efc77c663cb 1/1 column families, dataSize=289.77 KB heapSize=501.86 KB 2022-03-05 18:19:21,198 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=289.77 KB at sequenceid=9885 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/.tmp/e/4530f22bdb994e709b62ba6223bdeb44 2022-03-05 18:19:21,208 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/4530f22bdb994e709b62ba6223bdeb44, entries=2880, sequenceid=9885, filesize=30.4 K 2022-03-05 18:19:21,209 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~289.77 KB/296726, heapSize ~501.84 KB/513888, currentSize=0 B/0 for ccdacf4b414cb8d1bc548efc77c663cb in 49ms, sequenceid=9885, compaction requested=true 2022-03-05 18:19:21,209 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of ccdacf4b414cb8d1bc548efc77c663cb/e in aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb. 2022-03-05 18:19:21,209 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/48f4f75e5a23406097753ed12acd0c0f, hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/2f9e5226cb5e4f9594530b77b11591ec, hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/4530f22bdb994e709b62ba6223bdeb44] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/.tmp, totalSize=551.4 K 2022-03-05 18:19:21,294 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: ccdacf4b414cb8d1bc548efc77c663cb#e#compaction#147 average throughput is 62.49 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 18:19:21,332 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in ccdacf4b414cb8d1bc548efc77c663cb/e of ccdacf4b414cb8d1bc548efc77c663cb into 56fce020c0db425e9417e0b7597c1674(size=554.1 K), total size for store is 554.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 18:19:21,332 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., storeName=ccdacf4b414cb8d1bc548efc77c663cb/e, priority=13, startTime=1646504361209; duration=0sec 2022-03-05 18:19:22,324 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 7d1b53259f1d8280a47f7704f98f2c9f 1/1 column families, dataSize=677 KB heapSize=1.13 MB 2022-03-05 18:19:22,368 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=677 KB at sequenceid=16759 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/.tmp/e/b95ff47db3314206a3813462df45b6bf 2022-03-05 18:19:22,378 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/b95ff47db3314206a3813462df45b6bf, entries=6524, sequenceid=16759, filesize=64.6 K 2022-03-05 18:19:22,379 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~677 KB/693248, heapSize ~1.13 MB/1188792, currentSize=0 B/0 for 7d1b53259f1d8280a47f7704f98f2c9f in 56ms, sequenceid=16759, compaction requested=false 2022-03-05 18:21:08,909 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646503868852 with entries=1300, filesize=572.33 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646504468895 2022-03-05 18:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=11090, hits=8398, hitRatio=75.73%, , cachingAccesses=8439, cachingHits=8369, cachingHitsRatio=99.17%, evictions=5304, evicted=0, evictedPerRun=0.0 2022-03-05 18:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 18:23:42,174 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 0a764e51258807790269e46c55b99391 1/1 column families, dataSize=954.30 KB heapSize=1.67 MB 2022-03-05 18:23:42,369 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=954.30 KB at sequenceid=34206 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/.tmp/e/b06de88c4c5748758b481b4b449223d9 2022-03-05 18:23:42,379 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/b06de88c4c5748758b481b4b449223d9, entries=10272, sequenceid=34206, filesize=93.3 K 2022-03-05 18:23:42,380 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~954.30 KB/977200, heapSize ~1.67 MB/1756336, currentSize=0 B/0 for 0a764e51258807790269e46c55b99391 in 206ms, sequenceid=34206, compaction requested=true 2022-03-05 18:23:42,381 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 0a764e51258807790269e46c55b99391/e in aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391. 2022-03-05 18:23:42,381 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/4a9df5f62cc9402fa1608c87fb0e215c, hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/ce76b136df3a40a4979c15a721b5ea1c, hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/b06de88c4c5748758b481b4b449223d9] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/.tmp, totalSize=1.6 M 2022-03-05 18:23:42,609 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 0a764e51258807790269e46c55b99391#e#compaction#150 average throughput is 61.96 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 18:23:42,672 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 0a764e51258807790269e46c55b99391/e of 0a764e51258807790269e46c55b99391 into ce9ff36b3c384573bb8d5882287518ee(size=1.7 M), total size for store is 1.7 M. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 18:23:42,672 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391., storeName=0a764e51258807790269e46c55b99391/e, priority=13, startTime=1646504622381; duration=0sec 2022-03-05 18:23:48,930 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646504028869.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646504628911.meta 2022-03-05 18:23:48,930 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646504028869.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646504028869.meta 2022-03-05 18:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=11331, hits=8420, hitRatio=74.31%, , cachingAccesses=8461, cachingHits=8391, cachingHitsRatio=99.17%, evictions=5334, evicted=0, evictedPerRun=0.0 2022-03-05 18:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 18:31:08,952 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646504468895 with entries=1281, filesize=576.96 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646505068935 2022-03-05 18:31:08,953 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646500268603 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646500268603 2022-03-05 18:31:08,957 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646500868635 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646500868635 2022-03-05 18:31:08,965 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646501468673 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646501468673 2022-03-05 18:31:08,975 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646502068721 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646502068721 2022-03-05 18:31:08,980 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646502668762 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646502668762 2022-03-05 18:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=11353, hits=8442, hitRatio=74.36%, , cachingAccesses=8483, cachingHits=8413, cachingHitsRatio=99.17%, evictions=5364, evicted=0, evictedPerRun=0.0 2022-03-05 18:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-05 18:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 18:33:48,970 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646504628911.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646505228954.meta 2022-03-05 18:33:48,970 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646504628911.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646504628911.meta 2022-03-05 18:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=11381, hits=8470, hitRatio=74.42%, , cachingAccesses=8511, cachingHits=8441, cachingHitsRatio=99.18%, evictions=5394, evicted=0, evictedPerRun=0.0 2022-03-05 18:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 18:41:08,989 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646505068935 with entries=1292, filesize=572.19 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646505668976 2022-03-05 18:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=11403, hits=8492, hitRatio=74.47%, , cachingAccesses=8533, cachingHits=8463, cachingHitsRatio=99.18%, evictions=5424, evicted=0, evictedPerRun=0.0 2022-03-05 18:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 18:43:49,006 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646505228954.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646505828991.meta 2022-03-05 18:43:49,006 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646505228954.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646505228954.meta 2022-03-05 18:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=11425, hits=8514, hitRatio=74.52%, , cachingAccesses=8555, cachingHits=8485, cachingHitsRatio=99.18%, evictions=5454, evicted=0, evictedPerRun=0.0 2022-03-05 18:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 18:51:09,026 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646505668976 with entries=1284, filesize=566.66 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646506269011 2022-03-05 18:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=11447, hits=8536, hitRatio=74.57%, , cachingAccesses=8577, cachingHits=8507, cachingHitsRatio=99.18%, evictions=5484, evicted=0, evictedPerRun=0.0 2022-03-05 18:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 18:53:49,043 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646505828991.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646506429028.meta 2022-03-05 18:53:49,043 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646505828991.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646505828991.meta 2022-03-05 18:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=11469, hits=8558, hitRatio=74.62%, , cachingAccesses=8599, cachingHits=8529, cachingHitsRatio=99.19%, evictions=5514, evicted=0, evictedPerRun=0.0 2022-03-05 18:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 19:01:09,060 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646506269011 with entries=1283, filesize=566.33 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646506869048 2022-03-05 19:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=11491, hits=8580, hitRatio=74.67%, , cachingAccesses=8621, cachingHits=8551, cachingHitsRatio=99.19%, evictions=5544, evicted=0, evictedPerRun=0.0 2022-03-05 19:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 19:03:49,086 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646506429028.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646507029062.meta 2022-03-05 19:03:49,086 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646506429028.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646506429028.meta 2022-03-05 19:04:13,193 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb. because 9b121c22e18472616debd0996ee2a4cb/e has an old edit so flush to free WALs after random delay 83887 ms 2022-03-05 19:05:37,080 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 9b121c22e18472616debd0996ee2a4cb 1/1 column families, dataSize=556.85 KB heapSize=1.01 MB 2022-03-05 19:05:37,126 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=556.85 KB at sequenceid=26096 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/.tmp/e/d3e6051148a6466cb2dd62a8fa1848cb 2022-03-05 19:05:37,135 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/d3e6051148a6466cb2dd62a8fa1848cb, entries=6469, sequenceid=26096, filesize=67.3 K 2022-03-05 19:05:37,136 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~556.85 KB/570214, heapSize ~1.01 MB/1055616, currentSize=0 B/0 for 9b121c22e18472616debd0996ee2a4cb in 55ms, sequenceid=26096, compaction requested=false 2022-03-05 19:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=11513, hits=8602, hitRatio=74.72%, , cachingAccesses=8643, cachingHits=8573, cachingHitsRatio=99.19%, evictions=5574, evicted=0, evictedPerRun=0.0 2022-03-05 19:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 19:10:52,978 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5. because 7534b3b1812808d2d3c16342fbf347a5/e has an old edit so flush to free WALs after random delay 151758 ms 2022-03-05 19:11:09,107 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646506869048 with entries=1280, filesize=564.44 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646507469091 2022-03-05 19:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=11535, hits=8624, hitRatio=74.76%, , cachingAccesses=8665, cachingHits=8595, cachingHitsRatio=99.19%, evictions=5604, evicted=0, evictedPerRun=0.0 2022-03-05 19:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 19:12:32,976 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df. because 80a1ffe1dc2d8f258a6367d204dc64df/e has an old edit so flush to free WALs after random delay 67217 ms 2022-03-05 19:13:24,737 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 7534b3b1812808d2d3c16342fbf347a5 1/1 column families, dataSize=193.18 KB heapSize=361.51 KB 2022-03-05 19:13:24,786 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=193.18 KB at sequenceid=15381 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/.tmp/e/c84273595e1d42458c3e2b662a3799b9 2022-03-05 19:13:24,800 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/c84273595e1d42458c3e2b662a3799b9, entries=2270, sequenceid=15381, filesize=27.0 K 2022-03-05 19:13:24,801 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~193.18 KB/197812, heapSize ~361.49 KB/370168, currentSize=0 B/0 for 7534b3b1812808d2d3c16342fbf347a5 in 64ms, sequenceid=15381, compaction requested=true 2022-03-05 19:13:24,801 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 7534b3b1812808d2d3c16342fbf347a5/e in aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5. 2022-03-05 19:13:24,801 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/02617b84f0824268bcca4affb7792c63, hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/fb105568e4b74fae81af58505f164382, hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/c84273595e1d42458c3e2b662a3799b9] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/.tmp, totalSize=383.3 K 2022-03-05 19:13:24,863 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 7534b3b1812808d2d3c16342fbf347a5#e#compaction#153 average throughput is 58.49 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 19:13:24,900 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 7534b3b1812808d2d3c16342fbf347a5/e of 7534b3b1812808d2d3c16342fbf347a5 into b6ca65d9ba2b4579b3369cbc74749b85(size=375.6 K), total size for store is 375.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 19:13:24,900 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., storeName=7534b3b1812808d2d3c16342fbf347a5/e, priority=13, startTime=1646507604801; duration=0sec 2022-03-05 19:13:40,193 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 80a1ffe1dc2d8f258a6367d204dc64df 1/1 column families, dataSize=293.45 KB heapSize=548.40 KB 2022-03-05 19:13:40,234 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=293.45 KB at sequenceid=18889 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/.tmp/e/bfbc3040977545f4940ea1d3f9fbd142 2022-03-05 19:13:40,244 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/bfbc3040977545f4940ea1d3f9fbd142, entries=3479, sequenceid=18889, filesize=38.6 K 2022-03-05 19:13:40,245 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~293.45 KB/300493, heapSize ~548.38 KB/561544, currentSize=0 B/0 for 80a1ffe1dc2d8f258a6367d204dc64df in 52ms, sequenceid=18889, compaction requested=false 2022-03-05 19:13:49,122 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646507029062.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646507629110.meta 2022-03-05 19:13:49,122 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646507029062.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646507029062.meta 2022-03-05 19:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=11606, hits=8646, hitRatio=74.50%, , cachingAccesses=8687, cachingHits=8617, cachingHitsRatio=99.19%, evictions=5634, evicted=0, evictedPerRun=0.0 2022-03-05 19:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 19:17:43,354 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e. because 4333b489291b1f77671b8f2e78779a1e/__status_family___ has an old edit so flush to free WALs after random delay 74648 ms 2022-03-05 19:18:58,003 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 4333b489291b1f77671b8f2e78779a1e 1/1 column families, dataSize=7.74 KB heapSize=12.36 KB 2022-03-05 19:18:58,049 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=7.74 KB at sequenceid=1060 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/.tmp/__status_family___/33b4fd54b8c3457084d00cc3e9f079e3 2022-03-05 19:18:58,058 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/33b4fd54b8c3457084d00cc3e9f079e3, entries=3, sequenceid=1060, filesize=5.1 K 2022-03-05 19:18:58,059 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~7.74 KB/7928, heapSize ~12.34 KB/12640, currentSize=0 B/0 for 4333b489291b1f77671b8f2e78779a1e in 56ms, sequenceid=1060, compaction requested=true 2022-03-05 19:18:58,060 INFO [regionserver/cvp328:16201-longCompactions-0] regionserver.HRegion: Starting compaction of 4333b489291b1f77671b8f2e78779a1e/__status_family___ in __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e. 2022-03-05 19:18:58,060 INFO [regionserver/cvp328:16201-longCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/50f254634736454dbf0ade62b5b45b13, hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/e88261b886a346ada815ba08ffb7c345, hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/33b4fd54b8c3457084d00cc3e9f079e3] into tmpdir=hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/.tmp, totalSize=15.4 K 2022-03-05 19:18:58,073 INFO [regionserver/cvp328:16201-longCompactions-0] throttle.PressureAwareThroughputController: 4333b489291b1f77671b8f2e78779a1e#__status_family___#compaction#156 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 19:18:58,113 INFO [regionserver/cvp328:16201-longCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 4333b489291b1f77671b8f2e78779a1e/__status_family___ of 4333b489291b1f77671b8f2e78779a1e into 80a82e5ffa21463c9a0395c3e31555b7(size=5.2 K), total size for store is 5.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 19:18:58,113 INFO [regionserver/cvp328:16201-longCompactions-0] regionserver.CompactSplit: Completed compaction region=__test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., storeName=4333b489291b1f77671b8f2e78779a1e/__status_family___, priority=13, startTime=1646507938059; duration=0sec 2022-03-05 19:19:32,976 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. because 7d1b53259f1d8280a47f7704f98f2c9f/e has an old edit so flush to free WALs after random delay 159815 ms 2022-03-05 19:19:32,976 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb. because ccdacf4b414cb8d1bc548efc77c663cb/e has an old edit so flush to free WALs after random delay 124262 ms 2022-03-05 19:21:09,143 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646507469091 with entries=1288, filesize=563.49 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646508069127 2022-03-05 19:21:09,143 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646503268812 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646503268812 2022-03-05 19:21:37,238 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing ccdacf4b414cb8d1bc548efc77c663cb 1/1 column families, dataSize=296.38 KB heapSize=514.66 KB 2022-03-05 19:21:37,278 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=296.38 KB at sequenceid=10534 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/.tmp/e/038708b1dd8f453c89d1abae052688d0 2022-03-05 19:21:37,287 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/038708b1dd8f453c89d1abae052688d0, entries=2964, sequenceid=10534, filesize=31.2 K 2022-03-05 19:21:37,289 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~296.38 KB/303490, heapSize ~514.64 KB/526992, currentSize=0 B/0 for ccdacf4b414cb8d1bc548efc77c663cb in 51ms, sequenceid=10534, compaction requested=false 2022-03-05 19:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=11631, hits=8668, hitRatio=74.52%, , cachingAccesses=8709, cachingHits=8639, cachingHitsRatio=99.20%, evictions=5664, evicted=0, evictedPerRun=0.0 2022-03-05 19:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 19:22:12,790 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 7d1b53259f1d8280a47f7704f98f2c9f 1/1 column families, dataSize=664.75 KB heapSize=1.11 MB 2022-03-05 19:22:12,843 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=664.75 KB at sequenceid=17865 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/.tmp/e/0036ef44410b4536b514efc9b4cf5a50 2022-03-05 19:22:12,853 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/0036ef44410b4536b514efc9b4cf5a50, entries=6411, sequenceid=17865, filesize=64.1 K 2022-03-05 19:22:12,854 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~664.75 KB/680707, heapSize ~1.11 MB/1167672, currentSize=0 B/0 for 7d1b53259f1d8280a47f7704f98f2c9f in 64ms, sequenceid=17865, compaction requested=true 2022-03-05 19:22:12,854 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 7d1b53259f1d8280a47f7704f98f2c9f/e in aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. 2022-03-05 19:22:12,854 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/d5b6ae6ab3574842bd7776023a5ab09c, hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/b95ff47db3314206a3813462df45b6bf, hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/0036ef44410b4536b514efc9b4cf5a50] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/.tmp, totalSize=1.2 M 2022-03-05 19:22:13,032 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 7d1b53259f1d8280a47f7704f98f2c9f#e#compaction#159 average throughput is 62.03 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 19:22:13,081 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 7d1b53259f1d8280a47f7704f98f2c9f/e of 7d1b53259f1d8280a47f7704f98f2c9f into dab86dbf52c94f8e814d73f20f762314(size=1.3 M), total size for store is 1.3 M. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 19:22:13,081 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., storeName=7d1b53259f1d8280a47f7704f98f2c9f/e, priority=13, startTime=1646508132854; duration=0sec 2022-03-05 19:23:49,157 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646507629110.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646508229145.meta 2022-03-05 19:23:49,157 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646507629110.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646507629110.meta 2022-03-05 19:23:52,976 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391. because 0a764e51258807790269e46c55b99391/e has an old edit so flush to free WALs after random delay 49412 ms 2022-03-05 19:24:42,388 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 0a764e51258807790269e46c55b99391 1/1 column families, dataSize=904 KB heapSize=1.59 MB 2022-03-05 19:24:42,460 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=904 KB at sequenceid=36526 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/.tmp/e/bf018269b7f648569e875f95a048d9ce 2022-03-05 19:24:42,470 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/bf018269b7f648569e875f95a048d9ce, entries=9753, sequenceid=36526, filesize=88.0 K 2022-03-05 19:24:42,471 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~904 KB/925696, heapSize ~1.59 MB/1665560, currentSize=0 B/0 for 0a764e51258807790269e46c55b99391 in 84ms, sequenceid=36526, compaction requested=false 2022-03-05 19:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=11820, hits=8690, hitRatio=73.52%, , cachingAccesses=8731, cachingHits=8661, cachingHitsRatio=99.20%, evictions=5694, evicted=0, evictedPerRun=0.0 2022-03-05 19:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 19:31:09,187 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646508069127 with entries=1295, filesize=568.96 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646508669162 2022-03-05 19:31:09,188 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646503868852 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646503868852 2022-03-05 19:31:09,194 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646504468895 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646504468895 2022-03-05 19:31:09,201 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646505068935 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646505068935 2022-03-05 19:31:09,211 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646505668976 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646505668976 2022-03-05 19:31:09,215 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646506269011 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646506269011 2022-03-05 19:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=11842, hits=8712, hitRatio=73.57%, , cachingAccesses=8753, cachingHits=8683, cachingHitsRatio=99.20%, evictions=5724, evicted=0, evictedPerRun=0.0 2022-03-05 19:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-05 19:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 19:33:49,208 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646508229145.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646508829192.meta 2022-03-05 19:33:49,208 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646508229145.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646508229145.meta 2022-03-05 19:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=11870, hits=8740, hitRatio=73.63%, , cachingAccesses=8781, cachingHits=8711, cachingHitsRatio=99.20%, evictions=5754, evicted=0, evictedPerRun=0.0 2022-03-05 19:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 19:41:09,246 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646508669162 with entries=1276, filesize=565.58 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646509269216 2022-03-05 19:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=11892, hits=8762, hitRatio=73.68%, , cachingAccesses=8803, cachingHits=8733, cachingHitsRatio=99.20%, evictions=5784, evicted=0, evictedPerRun=0.0 2022-03-05 19:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 19:43:49,261 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646508829192.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646509429248.meta 2022-03-05 19:43:49,262 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646508829192.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646508829192.meta 2022-03-05 19:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=11914, hits=8784, hitRatio=73.73%, , cachingAccesses=8825, cachingHits=8755, cachingHitsRatio=99.21%, evictions=5814, evicted=0, evictedPerRun=0.0 2022-03-05 19:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 19:51:09,278 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646509269216 with entries=1297, filesize=564.81 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646509869266 2022-03-05 19:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=11936, hits=8806, hitRatio=73.78%, , cachingAccesses=8847, cachingHits=8777, cachingHitsRatio=99.21%, evictions=5844, evicted=0, evictedPerRun=0.0 2022-03-05 19:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 19:53:49,300 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646509429248.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646510029280.meta 2022-03-05 19:53:49,300 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646509429248.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646509429248.meta 2022-03-05 19:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=11958, hits=8828, hitRatio=73.83%, , cachingAccesses=8869, cachingHits=8799, cachingHitsRatio=99.21%, evictions=5874, evicted=0, evictedPerRun=0.0 2022-03-05 19:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 20:01:09,318 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646509869266 with entries=1291, filesize=569.69 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646510469305 2022-03-05 20:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=11980, hits=8850, hitRatio=73.87%, , cachingAccesses=8891, cachingHits=8821, cachingHitsRatio=99.21%, evictions=5904, evicted=0, evictedPerRun=0.0 2022-03-05 20:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 20:03:49,333 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646510029280.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646510629319.meta 2022-03-05 20:03:49,333 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646510029280.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646510029280.meta 2022-03-05 20:05:43,658 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb. because 9b121c22e18472616debd0996ee2a4cb/e has an old edit so flush to free WALs after random delay 220266 ms 2022-03-05 20:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=12002, hits=8872, hitRatio=73.92%, , cachingAccesses=8913, cachingHits=8843, cachingHitsRatio=99.21%, evictions=5934, evicted=0, evictedPerRun=0.0 2022-03-05 20:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 20:09:23,924 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 9b121c22e18472616debd0996ee2a4cb 1/1 column families, dataSize=575.59 KB heapSize=1.04 MB 2022-03-05 20:09:24,029 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=575.59 KB at sequenceid=27813 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/.tmp/e/f624ce22a7df457fb08ef5c3e9dc21cc 2022-03-05 20:09:24,039 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/f624ce22a7df457fb08ef5c3e9dc21cc, entries=6678, sequenceid=27813, filesize=70.3 K 2022-03-05 20:09:24,040 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~575.59 KB/589404, heapSize ~1.04 MB/1090312, currentSize=0 B/0 for 9b121c22e18472616debd0996ee2a4cb in 116ms, sequenceid=27813, compaction requested=true 2022-03-05 20:09:24,040 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 9b121c22e18472616debd0996ee2a4cb/e in aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb. 2022-03-05 20:09:24,040 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/b9f34279856044949a4426bd55b16f0e, hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/d3e6051148a6466cb2dd62a8fa1848cb, hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/f624ce22a7df457fb08ef5c3e9dc21cc] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/.tmp, totalSize=1.2 M 2022-03-05 20:09:24,241 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 9b121c22e18472616debd0996ee2a4cb#e#compaction#162 average throughput is 49.61 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 20:09:24,294 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 9b121c22e18472616debd0996ee2a4cb/e of 9b121c22e18472616debd0996ee2a4cb into f54e9addcf4b4594965df17962a6bb80(size=1.2 M), total size for store is 1.2 M. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 20:09:24,294 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., storeName=9b121c22e18472616debd0996ee2a4cb/e, priority=13, startTime=1646510964040; duration=0sec 2022-03-05 20:11:09,353 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646510469305 with entries=1279, filesize=555.07 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646511069341 2022-03-05 20:11:09,353 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646506869048 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646506869048 2022-03-05 20:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=12178, hits=8894, hitRatio=73.03%, , cachingAccesses=8935, cachingHits=8865, cachingHitsRatio=99.22%, evictions=5964, evicted=0, evictedPerRun=0.0 2022-03-05 20:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 20:13:33,290 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5. because 7534b3b1812808d2d3c16342fbf347a5/e has an old edit so flush to free WALs after random delay 255439 ms 2022-03-05 20:13:49,367 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646510629319.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646511229355.meta 2022-03-05 20:13:49,367 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646510629319.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646510629319.meta 2022-03-05 20:13:53,136 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df. because 80a1ffe1dc2d8f258a6367d204dc64df/e has an old edit so flush to free WALs after random delay 285747 ms 2022-03-05 20:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=12200, hits=8916, hitRatio=73.08%, , cachingAccesses=8957, cachingHits=8887, cachingHitsRatio=99.22%, evictions=5994, evicted=0, evictedPerRun=0.0 2022-03-05 20:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 20:17:48,729 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 7534b3b1812808d2d3c16342fbf347a5 1/1 column families, dataSize=184.92 KB heapSize=343.41 KB 2022-03-05 20:17:48,781 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=184.92 KB at sequenceid=16390 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/.tmp/e/c8affc11dcf446438a6c6ccd4302b92e 2022-03-05 20:17:48,790 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/c8affc11dcf446438a6c6ccd4302b92e, entries=2136, sequenceid=16390, filesize=25.8 K 2022-03-05 20:17:48,791 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~184.92 KB/189360, heapSize ~343.39 KB/351632, currentSize=0 B/0 for 7534b3b1812808d2d3c16342fbf347a5 in 62ms, sequenceid=16390, compaction requested=false 2022-03-05 20:18:38,883 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 80a1ffe1dc2d8f258a6367d204dc64df 1/1 column families, dataSize=307.60 KB heapSize=573.63 KB 2022-03-05 20:18:38,930 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=307.60 KB at sequenceid=20128 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/.tmp/e/6944919641c54b4593d7c8b603d2e8ca 2022-03-05 20:18:38,938 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/6944919641c54b4593d7c8b603d2e8ca, entries=3632, sequenceid=20128, filesize=40.2 K 2022-03-05 20:18:38,939 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~307.60 KB/314984, heapSize ~573.62 KB/587384, currentSize=0 B/0 for 80a1ffe1dc2d8f258a6367d204dc64df in 56ms, sequenceid=20128, compaction requested=true 2022-03-05 20:18:38,940 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 80a1ffe1dc2d8f258a6367d204dc64df/e in aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df. 2022-03-05 20:18:38,940 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/30b8b29a6df0406d8538da90cf648405, hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/bfbc3040977545f4940ea1d3f9fbd142, hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/6944919641c54b4593d7c8b603d2e8ca] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/.tmp, totalSize=686.9 K 2022-03-05 20:18:39,037 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 80a1ffe1dc2d8f258a6367d204dc64df#e#compaction#165 average throughput is 60.02 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 20:18:39,082 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 80a1ffe1dc2d8f258a6367d204dc64df/e of 80a1ffe1dc2d8f258a6367d204dc64df into 658b14da758a4bb4b1c539c23933edb4(size=684.0 K), total size for store is 684.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 20:18:39,082 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., storeName=80a1ffe1dc2d8f258a6367d204dc64df/e, priority=13, startTime=1646511518940; duration=0sec 2022-03-05 20:19:42,976 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e. because 4333b489291b1f77671b8f2e78779a1e/__status_family___ has an old edit so flush to free WALs after random delay 37327 ms 2022-03-05 20:20:20,303 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 4333b489291b1f77671b8f2e78779a1e 1/1 column families, dataSize=7.62 KB heapSize=12.16 KB 2022-03-05 20:20:20,354 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=7.62 KB at sequenceid=1125 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/.tmp/__status_family___/71401c867d60486ca2ba1d999525d29a 2022-03-05 20:20:20,364 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/71401c867d60486ca2ba1d999525d29a, entries=3, sequenceid=1125, filesize=5.1 K 2022-03-05 20:20:20,365 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~7.62 KB/7798, heapSize ~12.15 KB/12440, currentSize=0 B/0 for 4333b489291b1f77671b8f2e78779a1e in 62ms, sequenceid=1125, compaction requested=false 2022-03-05 20:21:09,387 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646511069341 with entries=1296, filesize=571.98 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646511669372 2022-03-05 20:21:09,388 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646507469091 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646507469091 2022-03-05 20:21:42,979 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb. because ccdacf4b414cb8d1bc548efc77c663cb/e has an old edit so flush to free WALs after random delay 112284 ms 2022-03-05 20:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=12306, hits=8938, hitRatio=72.63%, , cachingAccesses=8979, cachingHits=8909, cachingHitsRatio=99.22%, evictions=6024, evicted=0, evictedPerRun=0.0 2022-03-05 20:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 20:22:22,979 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. because 7d1b53259f1d8280a47f7704f98f2c9f/e has an old edit so flush to free WALs after random delay 183047 ms 2022-03-05 20:23:35,264 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing ccdacf4b414cb8d1bc548efc77c663cb 1/1 column families, dataSize=294.41 KB heapSize=511.61 KB 2022-03-05 20:23:35,371 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=294.41 KB at sequenceid=11175 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/.tmp/e/d16916e5acfa4240ad5c1a34563fd75d 2022-03-05 20:23:35,410 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/d16916e5acfa4240ad5c1a34563fd75d, entries=2949, sequenceid=11175, filesize=30.7 K 2022-03-05 20:23:35,415 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~294.41 KB/301471, heapSize ~511.59 KB/523872, currentSize=0 B/0 for ccdacf4b414cb8d1bc548efc77c663cb in 150ms, sequenceid=11175, compaction requested=true 2022-03-05 20:23:35,415 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of ccdacf4b414cb8d1bc548efc77c663cb/e in aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb. 2022-03-05 20:23:35,415 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/56fce020c0db425e9417e0b7597c1674, hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/038708b1dd8f453c89d1abae052688d0, hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/d16916e5acfa4240ad5c1a34563fd75d] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/.tmp, totalSize=616.0 K 2022-03-05 20:23:35,528 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: ccdacf4b414cb8d1bc548efc77c663cb#e#compaction#168 average throughput is 60.38 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 20:23:35,567 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in ccdacf4b414cb8d1bc548efc77c663cb/e of ccdacf4b414cb8d1bc548efc77c663cb into 91fbd17151ca4127a2353f3ef79ee1fc(size=619.6 K), total size for store is 619.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 20:23:35,567 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., storeName=ccdacf4b414cb8d1bc548efc77c663cb/e, priority=13, startTime=1646511815415; duration=0sec 2022-03-05 20:23:49,403 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646511229355.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646511829389.meta 2022-03-05 20:23:49,403 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646511229355.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646511229355.meta 2022-03-05 20:24:52,976 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391. because 0a764e51258807790269e46c55b99391/e has an old edit so flush to free WALs after random delay 260985 ms 2022-03-05 20:25:26,026 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 7d1b53259f1d8280a47f7704f98f2c9f 1/1 column families, dataSize=668.21 KB heapSize=1.12 MB 2022-03-05 20:25:26,093 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=668.21 KB at sequenceid=18985 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/.tmp/e/69894c587e0e4091bd01320488debecd 2022-03-05 20:25:26,102 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/69894c587e0e4091bd01320488debecd, entries=6440, sequenceid=18985, filesize=64.1 K 2022-03-05 20:25:26,103 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~668.21 KB/684242, heapSize ~1.12 MB/1173424, currentSize=0 B/0 for 7d1b53259f1d8280a47f7704f98f2c9f in 77ms, sequenceid=18985, compaction requested=false 2022-03-05 20:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=12410, hits=8960, hitRatio=72.20%, , cachingAccesses=9001, cachingHits=8931, cachingHitsRatio=99.22%, evictions=6054, evicted=0, evictedPerRun=0.0 2022-03-05 20:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 20:29:13,961 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 0a764e51258807790269e46c55b99391 1/1 column families, dataSize=950.67 KB heapSize=1.67 MB 2022-03-05 20:29:14,012 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=950.67 KB at sequenceid=38978 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/.tmp/e/ea0f3106f8e94244b70522e566099b12 2022-03-05 20:29:14,021 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/ea0f3106f8e94244b70522e566099b12, entries=10227, sequenceid=38978, filesize=93.2 K 2022-03-05 20:29:14,024 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~950.67 KB/973487, heapSize ~1.67 MB/1748848, currentSize=0 B/0 for 0a764e51258807790269e46c55b99391 in 63ms, sequenceid=38978, compaction requested=true 2022-03-05 20:29:14,025 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 0a764e51258807790269e46c55b99391/e in aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391. 2022-03-05 20:29:14,025 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/ce9ff36b3c384573bb8d5882287518ee, hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/bf018269b7f648569e875f95a048d9ce, hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/ea0f3106f8e94244b70522e566099b12] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/.tmp, totalSize=1.8 M 2022-03-05 20:29:14,284 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 0a764e51258807790269e46c55b99391#e#compaction#171 average throughput is 62.27 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 20:29:14,355 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 0a764e51258807790269e46c55b99391/e of 0a764e51258807790269e46c55b99391 into 91cea50eba41418299dce0c8c83b8fbe(size=1.9 M), total size for store is 1.9 M. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 20:29:14,355 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391., storeName=0a764e51258807790269e46c55b99391/e, priority=13, startTime=1646512154025; duration=0sec 2022-03-05 20:31:09,422 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646511669372 with entries=1290, filesize=565.30 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646512269410 2022-03-05 20:31:09,422 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646508069127 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646508069127 2022-03-05 20:31:09,429 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646508669162 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646508669162 2022-03-05 20:31:09,433 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646509269216 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646509269216 2022-03-05 20:31:09,440 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646509869266 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646509869266 2022-03-05 20:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=12681, hits=8982, hitRatio=70.83%, , cachingAccesses=9023, cachingHits=8953, cachingHitsRatio=99.22%, evictions=6084, evicted=0, evictedPerRun=0.0 2022-03-05 20:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-05 20:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 20:33:49,444 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646511829389.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646512429423.meta 2022-03-05 20:33:49,444 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646511829389.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646511829389.meta 2022-03-05 20:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=12709, hits=9010, hitRatio=70.89%, , cachingAccesses=9051, cachingHits=8981, cachingHitsRatio=99.23%, evictions=6114, evicted=0, evictedPerRun=0.0 2022-03-05 20:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 20:41:09,465 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646512269410 with entries=1274, filesize=574.40 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646512869454 2022-03-05 20:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=12731, hits=9032, hitRatio=70.94%, , cachingAccesses=9073, cachingHits=9003, cachingHitsRatio=99.23%, evictions=6144, evicted=0, evictedPerRun=0.0 2022-03-05 20:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 20:43:49,482 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646512429423.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646513029467.meta 2022-03-05 20:43:49,482 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646512429423.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646512429423.meta 2022-03-05 20:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=12753, hits=9054, hitRatio=71.00%, , cachingAccesses=9095, cachingHits=9025, cachingHitsRatio=99.23%, evictions=6174, evicted=0, evictedPerRun=0.0 2022-03-05 20:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 20:51:09,510 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646512869454 with entries=1297, filesize=568.27 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646513469497 2022-03-05 20:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=12775, hits=9076, hitRatio=71.05%, , cachingAccesses=9117, cachingHits=9047, cachingHitsRatio=99.23%, evictions=6204, evicted=0, evictedPerRun=0.0 2022-03-05 20:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 20:53:49,523 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646513029467.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646513629512.meta 2022-03-05 20:53:49,524 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646513029467.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646513029467.meta 2022-03-05 20:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=12797, hits=9098, hitRatio=71.09%, , cachingAccesses=9139, cachingHits=9069, cachingHitsRatio=99.23%, evictions=6234, evicted=0, evictedPerRun=0.0 2022-03-05 20:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 21:01:09,541 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646513469497 with entries=1283, filesize=568.35 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646514069528 2022-03-05 21:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=12819, hits=9120, hitRatio=71.14%, , cachingAccesses=9161, cachingHits=9091, cachingHitsRatio=99.24%, evictions=6264, evicted=0, evictedPerRun=0.0 2022-03-05 21:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 21:03:49,556 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646513629512.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646514229544.meta 2022-03-05 21:03:49,556 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646513629512.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646513629512.meta 2022-03-05 21:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=12841, hits=9142, hitRatio=71.19%, , cachingAccesses=9183, cachingHits=9113, cachingHitsRatio=99.24%, evictions=6294, evicted=0, evictedPerRun=0.0 2022-03-05 21:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 21:09:32,976 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb. because 9b121c22e18472616debd0996ee2a4cb/e has an old edit so flush to free WALs after random delay 48625 ms 2022-03-05 21:10:21,601 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 9b121c22e18472616debd0996ee2a4cb 1/1 column families, dataSize=556.86 KB heapSize=1.01 MB 2022-03-05 21:10:21,648 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=556.86 KB at sequenceid=29452 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/.tmp/e/b282ef900ccc4c83accb8b4f3cec7685 2022-03-05 21:10:21,657 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/b282ef900ccc4c83accb8b4f3cec7685, entries=6479, sequenceid=29452, filesize=67.4 K 2022-03-05 21:10:21,659 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~556.86 KB/570222, heapSize ~1.01 MB/1056248, currentSize=0 B/0 for 9b121c22e18472616debd0996ee2a4cb in 57ms, sequenceid=29452, compaction requested=false 2022-03-05 21:11:09,575 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646514069528 with entries=1295, filesize=557.15 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646514669563 2022-03-05 21:11:09,575 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646510469305 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646510469305 2022-03-05 21:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=12863, hits=9164, hitRatio=71.24%, , cachingAccesses=9205, cachingHits=9135, cachingHitsRatio=99.24%, evictions=6324, evicted=0, evictedPerRun=0.0 2022-03-05 21:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 21:13:49,591 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646514229544.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646514829579.meta 2022-03-05 21:13:49,591 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646514229544.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646514229544.meta 2022-03-05 21:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=12885, hits=9186, hitRatio=71.29%, , cachingAccesses=9227, cachingHits=9157, cachingHitsRatio=99.24%, evictions=6354, evicted=0, evictedPerRun=0.0 2022-03-05 21:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 21:18:03,497 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5. because 7534b3b1812808d2d3c16342fbf347a5/e has an old edit so flush to free WALs after random delay 71681 ms 2022-03-05 21:18:53,412 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df. because 80a1ffe1dc2d8f258a6367d204dc64df/e has an old edit so flush to free WALs after random delay 283502 ms 2022-03-05 21:19:15,179 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 7534b3b1812808d2d3c16342fbf347a5 1/1 column families, dataSize=179.48 KB heapSize=334.02 KB 2022-03-05 21:19:15,358 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=179.48 KB at sequenceid=17355 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/.tmp/e/93e8a76d01ef4dce82c27430a3fa7e2d 2022-03-05 21:19:15,380 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/93e8a76d01ef4dce82c27430a3fa7e2d, entries=2083, sequenceid=17355, filesize=25.0 K 2022-03-05 21:19:15,381 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~179.48 KB/183786, heapSize ~334.01 KB/342024, currentSize=0 B/0 for 7534b3b1812808d2d3c16342fbf347a5 in 202ms, sequenceid=17355, compaction requested=true 2022-03-05 21:19:15,381 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 7534b3b1812808d2d3c16342fbf347a5/e in aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5. 2022-03-05 21:19:15,382 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/b6ca65d9ba2b4579b3369cbc74749b85, hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/c8affc11dcf446438a6c6ccd4302b92e, hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/93e8a76d01ef4dce82c27430a3fa7e2d] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/.tmp, totalSize=426.4 K 2022-03-05 21:19:15,463 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 7534b3b1812808d2d3c16342fbf347a5#e#compaction#174 average throughput is 60.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 21:19:15,558 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 7534b3b1812808d2d3c16342fbf347a5/e of 7534b3b1812808d2d3c16342fbf347a5 into de8b36ea1a9a4e318f61f48d7ffa76e0(size=416.4 K), total size for store is 416.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 21:19:15,558 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., storeName=7534b3b1812808d2d3c16342fbf347a5/e, priority=13, startTime=1646515155381; duration=0sec 2022-03-05 21:20:43,625 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e. because 4333b489291b1f77671b8f2e78779a1e/__status_family___ has an old edit so flush to free WALs after random delay 280998 ms 2022-03-05 21:21:09,826 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646514669563 with entries=1282, filesize=570.56 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646515269815 2022-03-05 21:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=12961, hits=9208, hitRatio=71.04%, , cachingAccesses=9249, cachingHits=9179, cachingHitsRatio=99.24%, evictions=6384, evicted=0, evictedPerRun=0.0 2022-03-05 21:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 21:23:12,151 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2206 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 21:23:12,174 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 940 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-05 21:23:36,914 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 80a1ffe1dc2d8f258a6367d204dc64df 1/1 column families, dataSize=301.40 KB heapSize=560.55 KB 2022-03-05 21:23:37,027 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=301.40 KB at sequenceid=21367 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/.tmp/e/afa03dad12474cb1a541f00b02eeec0d 2022-03-05 21:23:37,039 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/afa03dad12474cb1a541f00b02eeec0d, entries=3537, sequenceid=21367, filesize=39.9 K 2022-03-05 21:23:37,040 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~301.40 KB/308636, heapSize ~560.54 KB/573992, currentSize=0 B/0 for 80a1ffe1dc2d8f258a6367d204dc64df in 126ms, sequenceid=21367, compaction requested=false 2022-03-05 21:23:42,976 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb. because ccdacf4b414cb8d1bc548efc77c663cb/e has an old edit so flush to free WALs after random delay 139077 ms 2022-03-05 21:23:49,858 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646514829579.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646515429832.meta 2022-03-05 21:23:49,859 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646514829579.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646514829579.meta 2022-03-05 21:25:24,624 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 4333b489291b1f77671b8f2e78779a1e 1/1 column families, dataSize=8.12 KB heapSize=12.95 KB 2022-03-05 21:25:24,755 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=8.12 KB at sequenceid=1193 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/.tmp/__status_family___/4363556facb14d319f711a8baa919dbc 2022-03-05 21:25:24,765 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/4363556facb14d319f711a8baa919dbc, entries=3, sequenceid=1193, filesize=5.1 K 2022-03-05 21:25:24,766 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~8.12 KB/8311, heapSize ~12.93 KB/13240, currentSize=0 B/0 for 4333b489291b1f77671b8f2e78779a1e in 142ms, sequenceid=1193, compaction requested=true 2022-03-05 21:25:24,766 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 4333b489291b1f77671b8f2e78779a1e/__status_family___ in __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e. 2022-03-05 21:25:24,767 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/80a82e5ffa21463c9a0395c3e31555b7, hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/71401c867d60486ca2ba1d999525d29a, hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/4363556facb14d319f711a8baa919dbc] into tmpdir=hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/.tmp, totalSize=15.4 K 2022-03-05 21:25:24,778 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 4333b489291b1f77671b8f2e78779a1e#__status_family___#compaction#177 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 21:25:24,812 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 4333b489291b1f77671b8f2e78779a1e/__status_family___ of 4333b489291b1f77671b8f2e78779a1e into 07aa1622edc749339d16e0114583b6cc(size=5.2 K), total size for store is 5.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 21:25:24,813 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=__test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., storeName=4333b489291b1f77671b8f2e78779a1e/__status_family___, priority=13, startTime=1646515524766; duration=0sec 2022-03-05 21:25:33,417 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. because 7d1b53259f1d8280a47f7704f98f2c9f/e has an old edit so flush to free WALs after random delay 128538 ms 2022-03-05 21:26:02,053 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing ccdacf4b414cb8d1bc548efc77c663cb 1/1 column families, dataSize=297.08 KB heapSize=516.22 KB 2022-03-05 21:26:02,181 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=297.08 KB at sequenceid=11828 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/.tmp/e/6b30c53bb90e4aaf80de0e1b22329f7d 2022-03-05 21:26:02,207 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/6b30c53bb90e4aaf80de0e1b22329f7d, entries=2976, sequenceid=11828, filesize=31.4 K 2022-03-05 21:26:02,209 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~297.08 KB/304210, heapSize ~516.20 KB/528592, currentSize=0 B/0 for ccdacf4b414cb8d1bc548efc77c663cb in 156ms, sequenceid=11828, compaction requested=false 2022-03-05 21:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=12986, hits=9230, hitRatio=71.08%, , cachingAccesses=9271, cachingHits=9201, cachingHitsRatio=99.24%, evictions=6414, evicted=0, evictedPerRun=0.0 2022-03-05 21:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 21:27:41,955 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 7d1b53259f1d8280a47f7704f98f2c9f 1/1 column families, dataSize=662.18 KB heapSize=1.11 MB 2022-03-05 21:27:42,092 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=662.18 KB at sequenceid=20090 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/.tmp/e/aee565c96eae46c8817929b5d1d90c22 2022-03-05 21:27:42,103 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/aee565c96eae46c8817929b5d1d90c22, entries=6401, sequenceid=20090, filesize=64.2 K 2022-03-05 21:27:42,104 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~662.18 KB/678074, heapSize ~1.11 MB/1164240, currentSize=0 B/0 for 7d1b53259f1d8280a47f7704f98f2c9f in 149ms, sequenceid=20090, compaction requested=true 2022-03-05 21:27:42,105 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 7d1b53259f1d8280a47f7704f98f2c9f/e in aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. 2022-03-05 21:27:42,105 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/dab86dbf52c94f8e814d73f20f762314, hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/69894c587e0e4091bd01320488debecd, hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/aee565c96eae46c8817929b5d1d90c22] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/.tmp, totalSize=1.4 M 2022-03-05 21:27:42,290 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 7d1b53259f1d8280a47f7704f98f2c9f#e#compaction#180 average throughput is 66.69 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 21:27:42,339 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 7d1b53259f1d8280a47f7704f98f2c9f/e of 7d1b53259f1d8280a47f7704f98f2c9f into 52d7483bd7f4491783ca5f6a0e2fad40(size=1.4 M), total size for store is 1.4 M. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 21:27:42,339 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., storeName=7d1b53259f1d8280a47f7704f98f2c9f/e, priority=13, startTime=1646515662105; duration=0sec 2022-03-05 21:29:23,488 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391. because 0a764e51258807790269e46c55b99391/e has an old edit so flush to free WALs after random delay 8391 ms 2022-03-05 21:29:31,879 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 0a764e51258807790269e46c55b99391 1/1 column families, dataSize=892.62 KB heapSize=1.57 MB 2022-03-05 21:29:31,948 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=892.62 KB at sequenceid=41275 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/.tmp/e/31b52465f4ec49f099036d2aab9e7c5b 2022-03-05 21:29:31,955 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/31b52465f4ec49f099036d2aab9e7c5b, entries=9605, sequenceid=41275, filesize=86.4 K 2022-03-05 21:29:31,956 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~892.62 KB/914043, heapSize ~1.57 MB/1642152, currentSize=0 B/0 for 0a764e51258807790269e46c55b99391 in 77ms, sequenceid=41275, compaction requested=false 2022-03-05 21:31:09,873 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646515269815 with entries=1286, filesize=560.42 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646515869863 2022-03-05 21:31:09,873 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646511069341 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646511069341 2022-03-05 21:31:09,878 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646511669372 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646511669372 2022-03-05 21:31:09,882 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646512269410 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646512269410 2022-03-05 21:31:09,889 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646512869454 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646512869454 2022-03-05 21:31:09,893 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646513469497 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646513469497 2022-03-05 21:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=13196, hits=9252, hitRatio=70.11%, , cachingAccesses=9293, cachingHits=9223, cachingHitsRatio=99.25%, evictions=6444, evicted=0, evictedPerRun=0.0 2022-03-05 21:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-05 21:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 21:33:49,901 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646515429832.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646516029875.meta 2022-03-05 21:33:49,902 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646515429832.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646515429832.meta 2022-03-05 21:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=13224, hits=9280, hitRatio=70.18%, , cachingAccesses=9321, cachingHits=9251, cachingHitsRatio=99.25%, evictions=6474, evicted=0, evictedPerRun=0.0 2022-03-05 21:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 21:41:09,917 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646515869863 with entries=1284, filesize=569.95 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646516469906 2022-03-05 21:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=333.90 KB, freeSize=306.87 MB, max=307.20 MB, blockCount=19, accesses=13334, hits=9378, hitRatio=70.33%, , cachingAccesses=9431, cachingHits=9349, cachingHitsRatio=99.13%, evictions=6504, evicted=0, evictedPerRun=0.0 2022-03-05 21:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 21:43:49,948 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646516029875.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646516629919.meta 2022-03-05 21:43:49,948 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646516029875.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646516029875.meta 2022-03-05 21:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=430.13 KB, freeSize=306.78 MB, max=307.20 MB, blockCount=23, accesses=13384, hits=9424, hitRatio=70.41%, , cachingAccesses=9481, cachingHits=9395, cachingHitsRatio=99.09%, evictions=6534, evicted=0, evictedPerRun=0.0 2022-03-05 21:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 21:51:09,963 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646516469906 with entries=1272, filesize=568.52 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646517069952 2022-03-05 21:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=524.67 KB, freeSize=306.69 MB, max=307.20 MB, blockCount=26, accesses=13439, hits=9476, hitRatio=70.51%, , cachingAccesses=9536, cachingHits=9447, cachingHitsRatio=99.07%, evictions=6564, evicted=0, evictedPerRun=0.0 2022-03-05 21:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 21:53:49,988 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646516629919.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646517229965.meta 2022-03-05 21:53:49,988 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646516629919.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646516629919.meta 2022-03-05 21:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=549.02 KB, freeSize=306.66 MB, max=307.20 MB, blockCount=27, accesses=13491, hits=9527, hitRatio=70.62%, , cachingAccesses=9588, cachingHits=9498, cachingHitsRatio=99.06%, evictions=6594, evicted=0, evictedPerRun=0.0 2022-03-05 21:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 22:01:10,015 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646517069952 with entries=1280, filesize=570.43 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646517669993 2022-03-05 22:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=549.02 KB, freeSize=306.66 MB, max=307.20 MB, blockCount=27, accesses=13529, hits=9565, hitRatio=70.70%, , cachingAccesses=9626, cachingHits=9536, cachingHitsRatio=99.07%, evictions=6624, evicted=0, evictedPerRun=0.0 2022-03-05 22:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 22:03:50,044 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646517229965.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646517830017.meta 2022-03-05 22:03:50,044 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646517229965.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646517229965.meta 2022-03-05 22:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=549.02 KB, freeSize=306.66 MB, max=307.20 MB, blockCount=27, accesses=13551, hits=9587, hitRatio=70.75%, , cachingAccesses=9648, cachingHits=9558, cachingHitsRatio=99.07%, evictions=6654, evicted=0, evictedPerRun=0.0 2022-03-05 22:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 22:10:23,597 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb. because 9b121c22e18472616debd0996ee2a4cb/e has an old edit so flush to free WALs after random delay 264028 ms 2022-03-05 22:11:10,072 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646517669993 with entries=1291, filesize=556.21 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646518270059 2022-03-05 22:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=549.02 KB, freeSize=306.66 MB, max=307.20 MB, blockCount=27, accesses=13573, hits=9609, hitRatio=70.79%, , cachingAccesses=9670, cachingHits=9580, cachingHitsRatio=99.07%, evictions=6684, evicted=0, evictedPerRun=0.0 2022-03-05 22:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 22:13:50,093 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646517830017.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646518430074.meta 2022-03-05 22:13:50,093 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646517830017.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646517830017.meta 2022-03-05 22:14:47,625 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 9b121c22e18472616debd0996ee2a4cb 1/1 column families, dataSize=587.15 KB heapSize=1.06 MB 2022-03-05 22:14:47,681 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=587.15 KB at sequenceid=31171 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/.tmp/e/257918a04cd148d1befacb89657c295d 2022-03-05 22:14:47,692 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/257918a04cd148d1befacb89657c295d, entries=6832, sequenceid=31171, filesize=72.1 K 2022-03-05 22:14:47,694 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~587.15 KB/601243, heapSize ~1.06 MB/1113856, currentSize=0 B/0 for 9b121c22e18472616debd0996ee2a4cb in 69ms, sequenceid=31171, compaction requested=true 2022-03-05 22:14:47,694 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 9b121c22e18472616debd0996ee2a4cb/e in aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb. 2022-03-05 22:14:47,694 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/f54e9addcf4b4594965df17962a6bb80, hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/b282ef900ccc4c83accb8b4f3cec7685, hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/257918a04cd148d1befacb89657c295d] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/.tmp, totalSize=1.4 M 2022-03-05 22:14:47,904 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 9b121c22e18472616debd0996ee2a4cb#e#compaction#183 average throughput is 54.16 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 22:14:47,971 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 9b121c22e18472616debd0996ee2a4cb/e of 9b121c22e18472616debd0996ee2a4cb into e94fbf53e02249aa8bd759cdec95252e(size=1.4 M), total size for store is 1.4 M. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 22:14:47,971 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., storeName=9b121c22e18472616debd0996ee2a4cb/e, priority=13, startTime=1646518487694; duration=0sec 2022-03-05 22:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=477.53 KB, freeSize=306.73 MB, max=307.20 MB, blockCount=23, accesses=13767, hits=9633, hitRatio=69.97%, , cachingAccesses=9692, cachingHits=9602, cachingHitsRatio=99.07%, evictions=6714, evicted=0, evictedPerRun=0.0 2022-03-05 22:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 22:19:23,097 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5. because 7534b3b1812808d2d3c16342fbf347a5/e has an old edit so flush to free WALs after random delay 60625 ms 2022-03-05 22:20:23,722 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 7534b3b1812808d2d3c16342fbf347a5 1/1 column families, dataSize=170.30 KB heapSize=315.45 KB 2022-03-05 22:20:23,828 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=170.30 KB at sequenceid=18306 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/.tmp/e/c07e4ce36cb44a7f9cb569caa5067f35 2022-03-05 22:20:23,838 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/c07e4ce36cb44a7f9cb569caa5067f35, entries=1955, sequenceid=18306, filesize=24.1 K 2022-03-05 22:20:23,841 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~170.30 KB/174383, heapSize ~315.43 KB/323000, currentSize=0 B/0 for 7534b3b1812808d2d3c16342fbf347a5 in 119ms, sequenceid=18306, compaction requested=false 2022-03-05 22:21:10,111 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646518270059 with entries=1285, filesize=564.17 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646518870098 2022-03-05 22:21:10,111 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646514069528 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646514069528 2022-03-05 22:21:10,115 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646514669563 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646514669563 2022-03-05 22:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=477.53 KB, freeSize=306.73 MB, max=307.20 MB, blockCount=23, accesses=13789, hits=9655, hitRatio=70.02%, , cachingAccesses=9714, cachingHits=9624, cachingHitsRatio=99.07%, evictions=6744, evicted=0, evictedPerRun=0.0 2022-03-05 22:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 22:23:50,128 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646518430074.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646519030113.meta 2022-03-05 22:23:50,129 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646518430074.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646518430074.meta 2022-03-05 22:23:52,976 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df. because 80a1ffe1dc2d8f258a6367d204dc64df/e has an old edit so flush to free WALs after random delay 47187 ms 2022-03-05 22:24:40,163 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 80a1ffe1dc2d8f258a6367d204dc64df 1/1 column families, dataSize=286.90 KB heapSize=534.87 KB 2022-03-05 22:24:40,210 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=286.90 KB at sequenceid=22529 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/.tmp/e/fdf5b344735e491aa76c806d5ed51453 2022-03-05 22:24:40,220 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/fdf5b344735e491aa76c806d5ed51453, entries=3385, sequenceid=22529, filesize=37.6 K 2022-03-05 22:24:40,221 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~286.90 KB/293781, heapSize ~534.85 KB/547688, currentSize=0 B/0 for 80a1ffe1dc2d8f258a6367d204dc64df in 58ms, sequenceid=22529, compaction requested=true 2022-03-05 22:24:40,222 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 80a1ffe1dc2d8f258a6367d204dc64df/e in aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df. 2022-03-05 22:24:40,222 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/658b14da758a4bb4b1c539c23933edb4, hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/afa03dad12474cb1a541f00b02eeec0d, hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/fdf5b344735e491aa76c806d5ed51453] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/.tmp, totalSize=761.5 K 2022-03-05 22:24:40,327 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 80a1ffe1dc2d8f258a6367d204dc64df#e#compaction#186 average throughput is 61.70 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 22:24:40,375 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 80a1ffe1dc2d8f258a6367d204dc64df/e of 80a1ffe1dc2d8f258a6367d204dc64df into 5cc8bcb5c56b4d99a6a6fdc3601a2625(size=757.3 K), total size for store is 757.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 22:24:40,375 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., storeName=80a1ffe1dc2d8f258a6367d204dc64df/e, priority=13, startTime=1646519080221; duration=0sec 2022-03-05 22:25:43,434 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e. because 4333b489291b1f77671b8f2e78779a1e/__status_family___ has an old edit so flush to free WALs after random delay 216246 ms 2022-03-05 22:26:13,433 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb. because ccdacf4b414cb8d1bc548efc77c663cb/e has an old edit so flush to free WALs after random delay 62303 ms 2022-03-05 22:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=450.31 KB, freeSize=306.76 MB, max=307.20 MB, blockCount=20, accesses=13905, hits=9678, hitRatio=69.60%, , cachingAccesses=9736, cachingHits=9646, cachingHitsRatio=99.08%, evictions=6774, evicted=0, evictedPerRun=0.0 2022-03-05 22:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 22:27:15,736 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing ccdacf4b414cb8d1bc548efc77c663cb 1/1 column families, dataSize=290.69 KB heapSize=504.88 KB 2022-03-05 22:27:15,801 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=290.69 KB at sequenceid=12446 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/.tmp/e/df1bac87013746be91209e576c65d143 2022-03-05 22:27:15,816 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/df1bac87013746be91209e576c65d143, entries=2906, sequenceid=12446, filesize=30.3 K 2022-03-05 22:27:15,817 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~290.69 KB/297669, heapSize ~504.87 KB/516984, currentSize=0 B/0 for ccdacf4b414cb8d1bc548efc77c663cb in 81ms, sequenceid=12446, compaction requested=true 2022-03-05 22:27:15,817 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of ccdacf4b414cb8d1bc548efc77c663cb/e in aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb. 2022-03-05 22:27:15,817 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/91fbd17151ca4127a2353f3ef79ee1fc, hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/6b30c53bb90e4aaf80de0e1b22329f7d, hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/df1bac87013746be91209e576c65d143] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/.tmp, totalSize=681.3 K 2022-03-05 22:27:15,923 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: ccdacf4b414cb8d1bc548efc77c663cb#e#compaction#188 average throughput is 62.08 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 22:27:15,979 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in ccdacf4b414cb8d1bc548efc77c663cb/e of ccdacf4b414cb8d1bc548efc77c663cb into f4c09ef4d5ff4b8d844cb7a114116cc8(size=692.7 K), total size for store is 692.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 22:27:15,979 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., storeName=ccdacf4b414cb8d1bc548efc77c663cb/e, priority=13, startTime=1646519235817; duration=0sec 2022-03-05 22:28:03,145 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. because 7d1b53259f1d8280a47f7704f98f2c9f/e has an old edit so flush to free WALs after random delay 260485 ms 2022-03-05 22:29:19,680 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 4333b489291b1f77671b8f2e78779a1e 1/1 column families, dataSize=8.00 KB heapSize=12.75 KB 2022-03-05 22:29:19,823 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=8.00 KB at sequenceid=1261 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/.tmp/__status_family___/4f51fdc54eb94bd583db09c4c731e531 2022-03-05 22:29:19,832 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/4f51fdc54eb94bd583db09c4c731e531, entries=3, sequenceid=1261, filesize=5.1 K 2022-03-05 22:29:19,833 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~8.00 KB/8188, heapSize ~12.73 KB/13040, currentSize=0 B/0 for 4333b489291b1f77671b8f2e78779a1e in 153ms, sequenceid=1261, compaction requested=false 2022-03-05 22:29:32,997 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391. because 0a764e51258807790269e46c55b99391/e has an old edit so flush to free WALs after random delay 72718 ms 2022-03-05 22:30:45,715 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 0a764e51258807790269e46c55b99391 1/1 column families, dataSize=901.43 KB heapSize=1.58 MB 2022-03-05 22:30:45,781 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=901.43 KB at sequenceid=43616 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/.tmp/e/3083f5dbe5654bd28f9d77189a925380 2022-03-05 22:30:45,799 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/3083f5dbe5654bd28f9d77189a925380, entries=9684, sequenceid=43616, filesize=86.4 K 2022-03-05 22:30:45,800 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~901.43 KB/923061, heapSize ~1.58 MB/1657136, currentSize=0 B/0 for 0a764e51258807790269e46c55b99391 in 85ms, sequenceid=43616, compaction requested=true 2022-03-05 22:30:45,800 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 0a764e51258807790269e46c55b99391/e in aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391. 2022-03-05 22:30:45,800 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/91cea50eba41418299dce0c8c83b8fbe, hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/31b52465f4ec49f099036d2aab9e7c5b, hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/3083f5dbe5654bd28f9d77189a925380] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/.tmp, totalSize=2.1 M 2022-03-05 22:30:46,111 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 0a764e51258807790269e46c55b99391#e#compaction#191 average throughput is 56.26 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 22:30:46,174 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 0a764e51258807790269e46c55b99391/e of 0a764e51258807790269e46c55b99391 into 8bfd69f9d1c3479d91942f5a1dc110d1(size=2.1 M), total size for store is 2.1 M. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 22:30:46,174 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391., storeName=0a764e51258807790269e46c55b99391/e, priority=13, startTime=1646519445800; duration=0sec 2022-03-05 22:31:10,145 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646518870098 with entries=1290, filesize=565.88 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646519470133 2022-03-05 22:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=448.05 KB, freeSize=306.76 MB, max=307.20 MB, blockCount=18, accesses=14296, hits=9703, hitRatio=67.87%, , cachingAccesses=9758, cachingHits=9668, cachingHitsRatio=99.08%, evictions=6804, evicted=0, evictedPerRun=0.0 2022-03-05 22:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-05 22:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 22:32:23,630 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 7d1b53259f1d8280a47f7704f98f2c9f 1/1 column families, dataSize=682.53 KB heapSize=1.14 MB 2022-03-05 22:32:23,687 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=682.53 KB at sequenceid=21230 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/.tmp/e/c2f26eab70be4a14b0903ff927e2af84 2022-03-05 22:32:23,697 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/c2f26eab70be4a14b0903ff927e2af84, entries=6577, sequenceid=21230, filesize=65.4 K 2022-03-05 22:32:23,698 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~682.53 KB/698915, heapSize ~1.14 MB/1198496, currentSize=0 B/0 for 7d1b53259f1d8280a47f7704f98f2c9f in 68ms, sequenceid=21230, compaction requested=false 2022-03-05 22:33:50,167 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646519030113.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646519630147.meta 2022-03-05 22:33:50,167 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646519030113.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646519030113.meta 2022-03-05 22:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=381.52 KB, freeSize=306.83 MB, max=307.20 MB, blockCount=13, accesses=14324, hits=9731, hitRatio=67.93%, , cachingAccesses=9786, cachingHits=9696, cachingHitsRatio=99.08%, evictions=6834, evicted=0, evictedPerRun=0.0 2022-03-05 22:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 22:41:10,182 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646519470133 with entries=1286, filesize=567.73 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646520070171 2022-03-05 22:41:10,182 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646515269815 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646515269815 2022-03-05 22:41:10,188 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646515869863 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646515869863 2022-03-05 22:41:10,194 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646516469906 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646516469906 2022-03-05 22:41:10,198 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646517069952 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646517069952 2022-03-05 22:41:10,203 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646517669993 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646517669993 2022-03-05 22:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=381.52 KB, freeSize=306.83 MB, max=307.20 MB, blockCount=13, accesses=14346, hits=9753, hitRatio=67.98%, , cachingAccesses=9808, cachingHits=9718, cachingHitsRatio=99.08%, evictions=6864, evicted=0, evictedPerRun=0.0 2022-03-05 22:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 22:43:50,208 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646519630147.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646520230183.meta 2022-03-05 22:43:50,209 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646519630147.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646519630147.meta 2022-03-05 22:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=381.52 KB, freeSize=306.83 MB, max=307.20 MB, blockCount=13, accesses=14368, hits=9775, hitRatio=68.03%, , cachingAccesses=9830, cachingHits=9740, cachingHitsRatio=99.08%, evictions=6894, evicted=0, evictedPerRun=0.0 2022-03-05 22:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 22:51:10,237 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646520070171 with entries=1276, filesize=567.77 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646520670215 2022-03-05 22:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=381.52 KB, freeSize=306.83 MB, max=307.20 MB, blockCount=13, accesses=14390, hits=9797, hitRatio=68.08%, , cachingAccesses=9852, cachingHits=9762, cachingHitsRatio=99.09%, evictions=6924, evicted=0, evictedPerRun=0.0 2022-03-05 22:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 22:53:50,261 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646520230183.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646520830239.meta 2022-03-05 22:53:50,261 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646520230183.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646520230183.meta 2022-03-05 22:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=381.52 KB, freeSize=306.83 MB, max=307.20 MB, blockCount=13, accesses=14412, hits=9819, hitRatio=68.13%, , cachingAccesses=9874, cachingHits=9784, cachingHitsRatio=99.09%, evictions=6954, evicted=0, evictedPerRun=0.0 2022-03-05 22:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 23:01:10,287 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646520670215 with entries=1281, filesize=563.04 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646521270269 2022-03-05 23:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=381.52 KB, freeSize=306.83 MB, max=307.20 MB, blockCount=13, accesses=14434, hits=9841, hitRatio=68.18%, , cachingAccesses=9896, cachingHits=9806, cachingHitsRatio=99.09%, evictions=6984, evicted=0, evictedPerRun=0.0 2022-03-05 23:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 23:03:50,307 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646520830239.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646521430289.meta 2022-03-05 23:03:50,307 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646520830239.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646520830239.meta 2022-03-05 23:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=381.52 KB, freeSize=306.83 MB, max=307.20 MB, blockCount=13, accesses=14456, hits=9863, hitRatio=68.23%, , cachingAccesses=9918, cachingHits=9828, cachingHitsRatio=99.09%, evictions=7014, evicted=0, evictedPerRun=0.0 2022-03-05 23:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 23:11:10,326 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646521270269 with entries=1273, filesize=563.91 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646521870312 2022-03-05 23:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=381.52 KB, freeSize=306.83 MB, max=307.20 MB, blockCount=13, accesses=14478, hits=9885, hitRatio=68.28%, , cachingAccesses=9940, cachingHits=9850, cachingHitsRatio=99.09%, evictions=7044, evicted=0, evictedPerRun=0.0 2022-03-05 23:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 23:13:50,346 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646521430289.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646522030327.meta 2022-03-05 23:13:50,346 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646521430289.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646521430289.meta 2022-03-05 23:14:52,979 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb. because 9b121c22e18472616debd0996ee2a4cb/e has an old edit so flush to free WALs after random delay 138444 ms 2022-03-05 23:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=381.52 KB, freeSize=306.83 MB, max=307.20 MB, blockCount=13, accesses=14500, hits=9907, hitRatio=68.32%, , cachingAccesses=9962, cachingHits=9872, cachingHitsRatio=99.10%, evictions=7074, evicted=0, evictedPerRun=0.0 2022-03-05 23:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 23:17:11,424 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 9b121c22e18472616debd0996ee2a4cb 1/1 column families, dataSize=571.85 KB heapSize=1.03 MB 2022-03-05 23:17:11,543 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=571.85 KB at sequenceid=32853 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/.tmp/e/b2d7cd5ba85f401593f6f1e55aa2be82 2022-03-05 23:17:11,551 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/b2d7cd5ba85f401593f6f1e55aa2be82, entries=6646, sequenceid=32853, filesize=68.7 K 2022-03-05 23:17:11,552 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~571.85 KB/585576, heapSize ~1.03 MB/1084200, currentSize=0 B/0 for 9b121c22e18472616debd0996ee2a4cb in 128ms, sequenceid=32853, compaction requested=false 2022-03-05 23:20:32,979 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5. because 7534b3b1812808d2d3c16342fbf347a5/e has an old edit so flush to free WALs after random delay 137426 ms 2022-03-05 23:21:10,364 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646521870312 with entries=1272, filesize=566.33 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646522470351 2022-03-05 23:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=381.52 KB, freeSize=306.83 MB, max=307.20 MB, blockCount=13, accesses=14522, hits=9929, hitRatio=68.37%, , cachingAccesses=9984, cachingHits=9894, cachingHitsRatio=99.10%, evictions=7104, evicted=0, evictedPerRun=0.0 2022-03-05 23:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 23:22:50,405 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 7534b3b1812808d2d3c16342fbf347a5 1/1 column families, dataSize=175.74 KB heapSize=325.87 KB 2022-03-05 23:22:50,449 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=175.74 KB at sequenceid=19277 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/.tmp/e/2020818b3f104ded9a01ebbd8df4a1b8 2022-03-05 23:22:50,457 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/2020818b3f104ded9a01ebbd8df4a1b8, entries=2022, sequenceid=19277, filesize=24.7 K 2022-03-05 23:22:50,458 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~175.74 KB/179958, heapSize ~325.85 KB/333672, currentSize=0 B/0 for 7534b3b1812808d2d3c16342fbf347a5 in 53ms, sequenceid=19277, compaction requested=true 2022-03-05 23:22:50,459 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 7534b3b1812808d2d3c16342fbf347a5/e in aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5. 2022-03-05 23:22:50,459 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/de8b36ea1a9a4e318f61f48d7ffa76e0, hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/c07e4ce36cb44a7f9cb569caa5067f35, hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/2020818b3f104ded9a01ebbd8df4a1b8] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/.tmp, totalSize=465.2 K 2022-03-05 23:22:50,531 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 7534b3b1812808d2d3c16342fbf347a5#e#compaction#195 average throughput is 61.38 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 23:22:50,572 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 7534b3b1812808d2d3c16342fbf347a5/e of 7534b3b1812808d2d3c16342fbf347a5 into c5a168411a5f4dfeb9f36e26681c3050(size=456.1 K), total size for store is 456.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 23:22:50,572 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., storeName=7534b3b1812808d2d3c16342fbf347a5/e, priority=13, startTime=1646522570458; duration=0sec 2022-03-05 23:23:50,380 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646522030327.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646522630366.meta 2022-03-05 23:23:50,380 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646522030327.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646522030327.meta 2022-03-05 23:24:53,329 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df. because 80a1ffe1dc2d8f258a6367d204dc64df/e has an old edit so flush to free WALs after random delay 131648 ms 2022-03-05 23:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=333.95 KB, freeSize=306.87 MB, max=307.20 MB, blockCount=10, accesses=14604, hits=9953, hitRatio=68.15%, , cachingAccesses=10006, cachingHits=9916, cachingHitsRatio=99.10%, evictions=7134, evicted=0, evictedPerRun=0.0 2022-03-05 23:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 23:27:04,977 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 80a1ffe1dc2d8f258a6367d204dc64df 1/1 column families, dataSize=295.98 KB heapSize=552.44 KB 2022-03-05 23:27:05,020 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=295.98 KB at sequenceid=23716 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/.tmp/e/4ee605230dfd4b2ab82cfd8884f4eee4 2022-03-05 23:27:05,029 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/4ee605230dfd4b2ab82cfd8884f4eee4, entries=3500, sequenceid=23716, filesize=39.4 K 2022-03-05 23:27:05,030 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~295.98 KB/303087, heapSize ~552.42 KB/565680, currentSize=0 B/0 for 80a1ffe1dc2d8f258a6367d204dc64df in 53ms, sequenceid=23716, compaction requested=false 2022-03-05 23:27:32,978 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb. because ccdacf4b414cb8d1bc548efc77c663cb/e has an old edit so flush to free WALs after random delay 139921 ms 2022-03-05 23:29:42,976 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e. because 4333b489291b1f77671b8f2e78779a1e/__status_family___ has an old edit so flush to free WALs after random delay 225844 ms 2022-03-05 23:29:52,899 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing ccdacf4b414cb8d1bc548efc77c663cb 1/1 column families, dataSize=293.10 KB heapSize=506.29 KB 2022-03-05 23:29:52,944 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=293.10 KB at sequenceid=13067 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/.tmp/e/925e10d2544448b683d55ff5cdc74888 2022-03-05 23:29:52,956 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/925e10d2544448b683d55ff5cdc74888, entries=2893, sequenceid=13067, filesize=30.1 K 2022-03-05 23:29:52,957 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~293.10 KB/300138, heapSize ~506.27 KB/518424, currentSize=0 B/0 for ccdacf4b414cb8d1bc548efc77c663cb in 58ms, sequenceid=13067, compaction requested=false 2022-03-05 23:30:53,227 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391. because 0a764e51258807790269e46c55b99391/e has an old edit so flush to free WALs after random delay 262570 ms 2022-03-05 23:31:10,401 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646522470351 with entries=1282, filesize=566.23 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646523070385 2022-03-05 23:31:10,401 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646518270059 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646518270059 2022-03-05 23:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=333.95 KB, freeSize=306.87 MB, max=307.20 MB, blockCount=10, accesses=14626, hits=9975, hitRatio=68.20%, , cachingAccesses=10028, cachingHits=9938, cachingHitsRatio=99.10%, evictions=7164, evicted=0, evictedPerRun=0.0 2022-03-05 23:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-05 23:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 23:32:32,976 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. because 7d1b53259f1d8280a47f7704f98f2c9f/e has an old edit so flush to free WALs after random delay 151711 ms 2022-03-05 23:33:28,820 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 4333b489291b1f77671b8f2e78779a1e 1/1 column families, dataSize=7.99 KB heapSize=12.75 KB 2022-03-05 23:33:28,876 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=7.99 KB at sequenceid=1328 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/.tmp/__status_family___/fb63046836c042c190049d1951a89e45 2022-03-05 23:33:28,886 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/fb63046836c042c190049d1951a89e45, entries=3, sequenceid=1328, filesize=5.1 K 2022-03-05 23:33:28,888 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~7.99 KB/8180, heapSize ~12.73 KB/13040, currentSize=0 B/0 for 4333b489291b1f77671b8f2e78779a1e in 68ms, sequenceid=1328, compaction requested=true 2022-03-05 23:33:28,889 INFO [regionserver/cvp328:16201-longCompactions-0] regionserver.HRegion: Starting compaction of 4333b489291b1f77671b8f2e78779a1e/__status_family___ in __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e. 2022-03-05 23:33:28,889 INFO [regionserver/cvp328:16201-longCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/07aa1622edc749339d16e0114583b6cc, hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/4f51fdc54eb94bd583db09c4c731e531, hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/fb63046836c042c190049d1951a89e45] into tmpdir=hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/.tmp, totalSize=15.5 K 2022-03-05 23:33:28,903 INFO [regionserver/cvp328:16201-longCompactions-0] throttle.PressureAwareThroughputController: 4333b489291b1f77671b8f2e78779a1e#__status_family___#compaction#199 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 23:33:28,941 INFO [regionserver/cvp328:16201-longCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 4333b489291b1f77671b8f2e78779a1e/__status_family___ of 4333b489291b1f77671b8f2e78779a1e into ba985a3c1fe847f8871e04a223b58252(size=5.2 K), total size for store is 5.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 23:33:28,941 INFO [regionserver/cvp328:16201-longCompactions-0] regionserver.CompactSplit: Completed compaction region=__test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., storeName=4333b489291b1f77671b8f2e78779a1e/__status_family___, priority=13, startTime=1646523208888; duration=0sec 2022-03-05 23:33:50,418 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646522630366.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646523230403.meta 2022-03-05 23:33:50,418 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646522630366.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646522630366.meta 2022-03-05 23:35:04,687 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 7d1b53259f1d8280a47f7704f98f2c9f 1/1 column families, dataSize=660.55 KB heapSize=1.10 MB 2022-03-05 23:35:04,754 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=660.55 KB at sequenceid=22333 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/.tmp/e/8c7a870921684728a0a6dd2cfa4d0aeb 2022-03-05 23:35:04,768 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/8c7a870921684728a0a6dd2cfa4d0aeb, entries=6348, sequenceid=22333, filesize=63.7 K 2022-03-05 23:35:04,769 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~660.55 KB/676399, heapSize ~1.10 MB/1158624, currentSize=0 B/0 for 7d1b53259f1d8280a47f7704f98f2c9f in 82ms, sequenceid=22333, compaction requested=true 2022-03-05 23:35:04,769 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 7d1b53259f1d8280a47f7704f98f2c9f/e in aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. 2022-03-05 23:35:04,769 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/52d7483bd7f4491783ca5f6a0e2fad40, hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/c2f26eab70be4a14b0903ff927e2af84, hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/8c7a870921684728a0a6dd2cfa4d0aeb] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/.tmp, totalSize=1.5 M 2022-03-05 23:35:04,977 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 7d1b53259f1d8280a47f7704f98f2c9f#e#compaction#201 average throughput is 68.73 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-05 23:35:05,047 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 7d1b53259f1d8280a47f7704f98f2c9f/e of 7d1b53259f1d8280a47f7704f98f2c9f into e03e0b809f6046ae90b8524a09f95cf8(size=1.6 M), total size for store is 1.6 M. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-05 23:35:05,047 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., storeName=7d1b53259f1d8280a47f7704f98f2c9f/e, priority=13, startTime=1646523304769; duration=0sec 2022-03-05 23:35:15,797 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 0a764e51258807790269e46c55b99391 1/1 column families, dataSize=959.96 KB heapSize=1.69 MB 2022-03-05 23:35:15,858 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=959.96 KB at sequenceid=46080 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/.tmp/e/275668bca2de449cb00a8e675099825a 2022-03-05 23:35:15,872 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/275668bca2de449cb00a8e675099825a, entries=10351, sequenceid=46080, filesize=93.8 K 2022-03-05 23:35:15,873 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~959.96 KB/982997, heapSize ~1.69 MB/1767576, currentSize=0 B/0 for 0a764e51258807790269e46c55b99391 in 76ms, sequenceid=46080, compaction requested=false 2022-03-05 23:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=14867, hits=10005, hitRatio=67.30%, , cachingAccesses=10056, cachingHits=9966, cachingHitsRatio=99.11%, evictions=7194, evicted=0, evictedPerRun=0.0 2022-03-05 23:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 23:41:10,434 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646523070385 with entries=1299, filesize=568.73 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646523670423 2022-03-05 23:41:10,434 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646518870098 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646518870098 2022-03-05 23:41:10,439 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646519470133 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646519470133 2022-03-05 23:41:10,445 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646520070171 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646520070171 2022-03-05 23:41:10,449 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646520670215 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646520670215 2022-03-05 23:41:10,453 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646521270269 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646521270269 2022-03-05 23:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=14889, hits=10027, hitRatio=67.35%, , cachingAccesses=10078, cachingHits=9988, cachingHitsRatio=99.11%, evictions=7224, evicted=0, evictedPerRun=0.0 2022-03-05 23:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 23:43:50,451 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646523230403.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646523830436.meta 2022-03-05 23:43:50,451 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646523230403.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646523230403.meta 2022-03-05 23:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=14911, hits=10049, hitRatio=67.39%, , cachingAccesses=10100, cachingHits=10010, cachingHitsRatio=99.11%, evictions=7254, evicted=0, evictedPerRun=0.0 2022-03-05 23:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 23:51:10,468 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646523670423 with entries=1282, filesize=569.23 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646524270455 2022-03-05 23:51:44,383 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=14933, hits=10071, hitRatio=67.44%, , cachingAccesses=10122, cachingHits=10032, cachingHitsRatio=99.11%, evictions=7284, evicted=0, evictedPerRun=0.0 2022-03-05 23:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-05 23:53:50,494 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646523830436.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646524430475.meta 2022-03-05 23:53:50,494 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646523830436.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646523830436.meta 2022-03-05 23:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=14955, hits=10093, hitRatio=67.49%, , cachingAccesses=10144, cachingHits=10054, cachingHitsRatio=99.11%, evictions=7314, evicted=0, evictedPerRun=0.0 2022-03-05 23:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 00:01:10,514 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646524270455 with entries=1287, filesize=562.32 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646524870499 2022-03-06 00:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=14977, hits=10115, hitRatio=67.54%, , cachingAccesses=10166, cachingHits=10076, cachingHitsRatio=99.11%, evictions=7344, evicted=0, evictedPerRun=0.0 2022-03-06 00:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 00:03:50,533 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646524430475.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646525030516.meta 2022-03-06 00:03:50,533 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646524430475.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646524430475.meta 2022-03-06 00:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=14999, hits=10137, hitRatio=67.58%, , cachingAccesses=10188, cachingHits=10098, cachingHitsRatio=99.12%, evictions=7374, evicted=0, evictedPerRun=0.0 2022-03-06 00:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 00:11:10,555 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646524870499 with entries=1275, filesize=563.11 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646525470538 2022-03-06 00:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=15021, hits=10159, hitRatio=67.63%, , cachingAccesses=10210, cachingHits=10120, cachingHitsRatio=99.12%, evictions=7404, evicted=0, evictedPerRun=0.0 2022-03-06 00:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 00:13:50,587 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646525030516.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646525630557.meta 2022-03-06 00:13:50,587 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646525030516.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646525030516.meta 2022-03-06 00:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=15043, hits=10181, hitRatio=67.68%, , cachingAccesses=10232, cachingHits=10142, cachingHitsRatio=99.12%, evictions=7434, evicted=0, evictedPerRun=0.0 2022-03-06 00:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 00:17:12,976 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb. because 9b121c22e18472616debd0996ee2a4cb/e has an old edit so flush to free WALs after random delay 50363 ms 2022-03-06 00:18:03,339 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 9b121c22e18472616debd0996ee2a4cb 1/1 column families, dataSize=554.44 KB heapSize=1.00 MB 2022-03-06 00:18:03,395 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=554.44 KB at sequenceid=34494 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/.tmp/e/e52d894ec67341afa32c4cdc4388e1d3 2022-03-06 00:18:03,404 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/e52d894ec67341afa32c4cdc4388e1d3, entries=6441, sequenceid=34494, filesize=67.6 K 2022-03-06 00:18:03,405 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~554.44 KB/567744, heapSize ~1.00 MB/1050872, currentSize=0 B/0 for 9b121c22e18472616debd0996ee2a4cb in 67ms, sequenceid=34494, compaction requested=true 2022-03-06 00:18:03,406 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 9b121c22e18472616debd0996ee2a4cb/e in aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb. 2022-03-06 00:18:03,406 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/e94fbf53e02249aa8bd759cdec95252e, hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/b2d7cd5ba85f401593f6f1e55aa2be82, hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/e52d894ec67341afa32c4cdc4388e1d3] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/.tmp, totalSize=1.5 M 2022-03-06 00:18:03,652 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 9b121c22e18472616debd0996ee2a4cb#e#compaction#204 average throughput is 51.09 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-06 00:18:03,701 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 9b121c22e18472616debd0996ee2a4cb/e of 9b121c22e18472616debd0996ee2a4cb into 3d547b9baa16448a9a317216f9f19370(size=1.5 M), total size for store is 1.5 M. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-06 00:18:03,701 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., storeName=9b121c22e18472616debd0996ee2a4cb/e, priority=13, startTime=1646525883405; duration=0sec 2022-03-06 00:21:10,603 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646525470538 with entries=1275, filesize=569.51 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646526070592 2022-03-06 00:21:10,603 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646521870312 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646521870312 2022-03-06 00:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=15255, hits=10203, hitRatio=66.88%, , cachingAccesses=10254, cachingHits=10164, cachingHitsRatio=99.12%, evictions=7464, evicted=0, evictedPerRun=0.0 2022-03-06 00:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 00:23:03,065 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5. because 7534b3b1812808d2d3c16342fbf347a5/e has an old edit so flush to free WALs after random delay 67215 ms 2022-03-06 00:23:50,621 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646525630557.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646526230605.meta 2022-03-06 00:23:50,621 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646525630557.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646525630557.meta 2022-03-06 00:24:10,281 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 7534b3b1812808d2d3c16342fbf347a5 1/1 column families, dataSize=181.34 KB heapSize=337.88 KB 2022-03-06 00:24:10,333 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=181.34 KB at sequenceid=20243 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/.tmp/e/c55e42e7302d4c0ca206483e7855cf25 2022-03-06 00:24:10,345 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/c55e42e7302d4c0ca206483e7855cf25, entries=2111, sequenceid=20243, filesize=25.5 K 2022-03-06 00:24:10,346 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~181.34 KB/185690, heapSize ~337.86 KB/345968, currentSize=0 B/0 for 7534b3b1812808d2d3c16342fbf347a5 in 65ms, sequenceid=20243, compaction requested=false 2022-03-06 00:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=15277, hits=10225, hitRatio=66.93%, , cachingAccesses=10276, cachingHits=10186, cachingHitsRatio=99.12%, evictions=7494, evicted=0, evictedPerRun=0.0 2022-03-06 00:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 00:27:12,977 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df. because 80a1ffe1dc2d8f258a6367d204dc64df/e has an old edit so flush to free WALs after random delay 271518 ms 2022-03-06 00:30:02,976 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb. because ccdacf4b414cb8d1bc548efc77c663cb/e has an old edit so flush to free WALs after random delay 135412 ms 2022-03-06 00:31:10,706 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646526070592 with entries=1277, filesize=559.69 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646526670627 2022-03-06 00:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=15299, hits=10247, hitRatio=66.98%, , cachingAccesses=10298, cachingHits=10208, cachingHitsRatio=99.13%, evictions=7524, evicted=0, evictedPerRun=0.0 2022-03-06 00:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-06 00:31:44,495 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 80a1ffe1dc2d8f258a6367d204dc64df 1/1 column families, dataSize=310.83 KB heapSize=580.71 KB 2022-03-06 00:31:44,623 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=310.83 KB at sequenceid=24946 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/.tmp/e/80845f0284424b94b899cc21ae825dfc 2022-03-06 00:31:44,633 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/80845f0284424b94b899cc21ae825dfc, entries=3684, sequenceid=24946, filesize=41.3 K 2022-03-06 00:31:44,634 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~310.83 KB/318290, heapSize ~580.70 KB/594632, currentSize=0 B/0 for 80a1ffe1dc2d8f258a6367d204dc64df in 139ms, sequenceid=24946, compaction requested=true 2022-03-06 00:31:44,635 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 80a1ffe1dc2d8f258a6367d204dc64df/e in aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df. 2022-03-06 00:31:44,635 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/5cc8bcb5c56b4d99a6a6fdc3601a2625, hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/4ee605230dfd4b2ab82cfd8884f4eee4, hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/80845f0284424b94b899cc21ae825dfc] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/.tmp, totalSize=838.0 K 2022-03-06 00:31:44,751 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 80a1ffe1dc2d8f258a6367d204dc64df#e#compaction#207 average throughput is 59.70 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-06 00:31:45,042 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 80a1ffe1dc2d8f258a6367d204dc64df/e of 80a1ffe1dc2d8f258a6367d204dc64df into 2f56f68748df4891a59c60e076b28aab(size=834.6 K), total size for store is 834.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-06 00:31:45,042 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., storeName=80a1ffe1dc2d8f258a6367d204dc64df/e, priority=13, startTime=1646526704634; duration=0sec 2022-03-06 00:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 00:32:18,388 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing ccdacf4b414cb8d1bc548efc77c663cb 1/1 column families, dataSize=293.01 KB heapSize=508.25 KB 2022-03-06 00:32:18,509 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=293.01 KB at sequenceid=13703 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/.tmp/e/dab578997a974f2fb01ae44bd3428c58 2022-03-06 00:32:18,520 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/dab578997a974f2fb01ae44bd3428c58, entries=2922, sequenceid=13703, filesize=30.8 K 2022-03-06 00:32:18,522 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~293.01 KB/300043, heapSize ~508.23 KB/520432, currentSize=0 B/0 for ccdacf4b414cb8d1bc548efc77c663cb in 134ms, sequenceid=13703, compaction requested=true 2022-03-06 00:32:18,522 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of ccdacf4b414cb8d1bc548efc77c663cb/e in aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb. 2022-03-06 00:32:18,522 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/f4c09ef4d5ff4b8d844cb7a114116cc8, hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/925e10d2544448b683d55ff5cdc74888, hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/dab578997a974f2fb01ae44bd3428c58] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/.tmp, totalSize=753.6 K 2022-03-06 00:32:18,648 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: ccdacf4b414cb8d1bc548efc77c663cb#e#compaction#209 average throughput is 54.44 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-06 00:32:19,058 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in ccdacf4b414cb8d1bc548efc77c663cb/e of ccdacf4b414cb8d1bc548efc77c663cb into 4e984961cc7b4f43a4b9dc57065e19e5(size=757.0 K), total size for store is 757.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-06 00:32:19,058 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., storeName=ccdacf4b414cb8d1bc548efc77c663cb/e, priority=13, startTime=1646526738522; duration=0sec 2022-03-06 00:33:42,977 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e. because 4333b489291b1f77671b8f2e78779a1e/__status_family___ has an old edit so flush to free WALs after random delay 297186 ms 2022-03-06 00:33:50,723 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646526230605.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646526830708.meta 2022-03-06 00:33:50,723 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646526230605.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646526230605.meta 2022-03-06 00:35:12,977 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. because 7d1b53259f1d8280a47f7704f98f2c9f/e has an old edit so flush to free WALs after random delay 68836 ms 2022-03-06 00:35:22,976 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391. because 0a764e51258807790269e46c55b99391/e has an old edit so flush to free WALs after random delay 215079 ms 2022-03-06 00:36:21,813 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 7d1b53259f1d8280a47f7704f98f2c9f 1/1 column families, dataSize=643.27 KB heapSize=1.08 MB 2022-03-06 00:36:21,886 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=643.27 KB at sequenceid=23400 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/.tmp/e/160d8782509444d1918349e6a944173b 2022-03-06 00:36:21,936 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/160d8782509444d1918349e6a944173b, entries=6190, sequenceid=23400, filesize=61.9 K 2022-03-06 00:36:21,937 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~643.27 KB/658709, heapSize ~1.08 MB/1128912, currentSize=0 B/0 for 7d1b53259f1d8280a47f7704f98f2c9f in 124ms, sequenceid=23400, compaction requested=false 2022-03-06 00:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=15532, hits=10275, hitRatio=66.15%, , cachingAccesses=10326, cachingHits=10236, cachingHitsRatio=99.13%, evictions=7554, evicted=0, evictedPerRun=0.0 2022-03-06 00:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 00:38:40,163 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 4333b489291b1f77671b8f2e78779a1e 1/1 column families, dataSize=8.24 KB heapSize=13.14 KB 2022-03-06 00:38:40,202 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=8.24 KB at sequenceid=1398 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/.tmp/__status_family___/e8c0224421f6444cba6d00b24f1e4a8a 2022-03-06 00:38:40,210 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/e8c0224421f6444cba6d00b24f1e4a8a, entries=3, sequenceid=1398, filesize=5.1 K 2022-03-06 00:38:40,211 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~8.24 KB/8435, heapSize ~13.13 KB/13440, currentSize=0 B/0 for 4333b489291b1f77671b8f2e78779a1e in 48ms, sequenceid=1398, compaction requested=false 2022-03-06 00:38:58,055 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 0a764e51258807790269e46c55b99391 1/1 column families, dataSize=938.35 KB heapSize=1.65 MB 2022-03-06 00:38:58,221 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=938.35 KB at sequenceid=48504 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/.tmp/e/2a9208ab300444159b75a88c795689e2 2022-03-06 00:38:58,231 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/2a9208ab300444159b75a88c795689e2, entries=10086, sequenceid=48504, filesize=92.3 K 2022-03-06 00:38:58,233 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~938.35 KB/960868, heapSize ~1.65 MB/1725392, currentSize=0 B/0 for 0a764e51258807790269e46c55b99391 in 178ms, sequenceid=48504, compaction requested=true 2022-03-06 00:38:58,233 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 0a764e51258807790269e46c55b99391/e in aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391. 2022-03-06 00:38:58,233 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/8bfd69f9d1c3479d91942f5a1dc110d1, hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/275668bca2de449cb00a8e675099825a, hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/2a9208ab300444159b75a88c795689e2] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/.tmp, totalSize=2.3 M 2022-03-06 00:38:58,574 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 0a764e51258807790269e46c55b99391#e#compaction#213 average throughput is 57.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-06 00:38:58,731 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 0a764e51258807790269e46c55b99391/e of 0a764e51258807790269e46c55b99391 into 53326ff0062d4571b8205cad318872c6(size=2.3 M), total size for store is 2.3 M. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-06 00:38:58,731 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391., storeName=0a764e51258807790269e46c55b99391/e, priority=13, startTime=1646527138233; duration=0sec 2022-03-06 00:41:10,742 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646526670627 with entries=1298, filesize=568.24 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646527270729 2022-03-06 00:41:10,742 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646522470351 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646522470351 2022-03-06 00:41:10,747 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646523070385 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646523070385 2022-03-06 00:41:10,752 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646523670423 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646523670423 2022-03-06 00:41:10,758 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646524270455 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646524270455 2022-03-06 00:41:10,764 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646524870499 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646524870499 2022-03-06 00:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=15864, hits=10297, hitRatio=64.91%, , cachingAccesses=10348, cachingHits=10258, cachingHitsRatio=99.13%, evictions=7584, evicted=0, evictedPerRun=0.0 2022-03-06 00:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 00:43:50,767 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646526830708.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646527430744.meta 2022-03-06 00:43:50,768 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646526830708.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646526830708.meta 2022-03-06 00:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=15886, hits=10319, hitRatio=64.96%, , cachingAccesses=10370, cachingHits=10280, cachingHitsRatio=99.13%, evictions=7614, evicted=0, evictedPerRun=0.0 2022-03-06 00:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 00:51:10,782 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646527270729 with entries=1294, filesize=571.82 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646527870772 2022-03-06 00:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=15908, hits=10341, hitRatio=65.01%, , cachingAccesses=10392, cachingHits=10302, cachingHitsRatio=99.13%, evictions=7644, evicted=0, evictedPerRun=0.0 2022-03-06 00:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 00:53:50,803 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646527430744.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646528030784.meta 2022-03-06 00:53:50,803 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646527430744.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646527430744.meta 2022-03-06 00:54:10,009 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2661 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 00:54:10,009 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1307 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 00:54:10,009 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1300 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 00:54:41,966 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 00:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=15930, hits=10363, hitRatio=65.05%, , cachingAccesses=10414, cachingHits=10324, cachingHitsRatio=99.14%, evictions=7674, evicted=0, evictedPerRun=0.0 2022-03-06 00:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 00:56:55,592 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1303 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 00:56:55,592 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 675 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 00:56:58,086 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2174 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 00:56:58,086 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2121 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 00:57:29,489 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 730 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 00:57:29,489 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 556 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 00:57:29,489 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 555 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 00:57:31,220 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1090 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 00:57:31,220 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 827 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 00:57:31,220 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 799 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 00:57:32,810 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1590 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 00:57:32,810 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1587 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 00:57:32,810 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1490 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 00:58:01,978 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 986 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 01:00:07,295 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1999ms No GCs detected 2022-03-06 01:00:09,861 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5663 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 01:00:11,407 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4108 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 01:00:11,393 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3597ms No GCs detected 2022-03-06 01:00:12,528 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 604 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 01:00:22,092 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1518ms No GCs detected 2022-03-06 01:00:22,941 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 833 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 01:00:22,942 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 533 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 01:01:12,181 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646527870772 with entries=1294, filesize=568.70 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646528472099 2022-03-06 01:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=15952, hits=10385, hitRatio=65.10%, , cachingAccesses=10436, cachingHits=10346, cachingHitsRatio=99.14%, evictions=7704, evicted=0, evictedPerRun=0.0 2022-03-06 01:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 01:03:04,612 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1177 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:03:14,416 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3099 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:03:14,416 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3042 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:03:14,434 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2666 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:03:14,434 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1203 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:03:16,125 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 01:03:28,271 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2317 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:03:28,271 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2298 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:03:34,311 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 876 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:03:37,587 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3313 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:03:40,562 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 708 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:03:53,398 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646528030784.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646528632189.meta 2022-03-06 01:03:53,399 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646528030784.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646528030784.meta 2022-03-06 01:04:01,874 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 907 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:04:33,038 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2054 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:04:33,039 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 863 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:04:33,039 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 737 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:04:59,909 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 596 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:05:33,039 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3019 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:05:33,039 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3008 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:05:33,040 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2607 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:05:33,040 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2581 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:05:33,040 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2310 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:05:33,040 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1311 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:05:47,924 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 01:06:30,520 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 592 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:06:30,520 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 534 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=15974, hits=10407, hitRatio=65.15%, , cachingAccesses=10458, cachingHits=10368, cachingHitsRatio=99.14%, evictions=7734, evicted=0, evictedPerRun=0.0 2022-03-06 01:06:47,957 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 700 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:06:47,957 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 575 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 01:07:02,706 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 555 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:07:02,706 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 554 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:07:54,005 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1588 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:07:54,005 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 751 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:08:12,316 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2455 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:08:24,322 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1914 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:09:01,867 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5949 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:09:01,867 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5941 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:09:01,867 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5824 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:09:01,867 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3975 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:09:01,867 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3960 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:09:01,867 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3775 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:09:01,867 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3271 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:09:01,867 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2438 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:09:01,867 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2438 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:09:01,867 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2430 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:09:01,867 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2425 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:09:01,867 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2040 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:09:01,867 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1075 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:09:01,867 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 817 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:10:28,465 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 526 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:10:28,465 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 513 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:11:03,151 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 693 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:11:03,540 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 727 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:11:13,594 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646528472099 with entries=1293, filesize=567.92 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646529073585 2022-03-06 01:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=15996, hits=10429, hitRatio=65.20%, , cachingAccesses=10480, cachingHits=10390, cachingHitsRatio=99.14%, evictions=7764, evicted=0, evictedPerRun=0.0 2022-03-06 01:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 01:13:33,772 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1581 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:13:33,773 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1581 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:13:33,773 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 975 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:13:53,779 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646528632189.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646529233763.meta 2022-03-06 01:13:53,779 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646528632189.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646528632189.meta 2022-03-06 01:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=16018, hits=10451, hitRatio=65.25%, , cachingAccesses=10502, cachingHits=10412, cachingHitsRatio=99.14%, evictions=7794, evicted=0, evictedPerRun=0.0 2022-03-06 01:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 01:18:07,972 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 813 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:18:13,257 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb. because 9b121c22e18472616debd0996ee2a4cb/e has an old edit so flush to free WALs after random delay 64692 ms 2022-03-06 01:19:17,950 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 9b121c22e18472616debd0996ee2a4cb 1/1 column families, dataSize=559.58 KB heapSize=1.01 MB 2022-03-06 01:19:18,002 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=559.58 KB at sequenceid=36151 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/.tmp/e/f0dc5dcf7ae0494dbb40102a5131bcc0 2022-03-06 01:19:18,008 INFO [MemStoreFlusher.1] retry.RetryInvocationHandler: doop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:433) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getBlockLocations over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 01:19:18,014 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/f0dc5dcf7ae0494dbb40102a5131bcc0, entries=6490, sequenceid=36151, filesize=68.6 K 2022-03-06 01:19:18,015 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~559.58 KB/573005, heapSize ~1.01 MB/1059904, currentSize=0 B/0 for 9b121c22e18472616debd0996ee2a4cb in 66ms, sequenceid=36151, compaction requested=false 2022-03-06 01:19:38,918 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 701 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:21:13,800 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646529073585 with entries=1288, filesize=564.79 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646529673788 2022-03-06 01:21:13,800 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646525470538 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646525470538 2022-03-06 01:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=16040, hits=10473, hitRatio=65.29%, , cachingAccesses=10524, cachingHits=10434, cachingHitsRatio=99.14%, evictions=7824, evicted=0, evictedPerRun=0.0 2022-03-06 01:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 01:23:53,818 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646529233763.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646529833809.meta 2022-03-06 01:23:53,818 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646529233763.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646529233763.meta 2022-03-06 01:24:22,998 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5. because 7534b3b1812808d2d3c16342fbf347a5/e has an old edit so flush to free WALs after random delay 26026 ms 2022-03-06 01:24:49,024 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 7534b3b1812808d2d3c16342fbf347a5 1/1 column families, dataSize=171.20 KB heapSize=317.65 KB 2022-03-06 01:24:50,004 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=171.20 KB at sequenceid=21197 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/.tmp/e/0cee6657467c4549b82e8d77f05432a6 2022-03-06 01:24:50,016 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/0cee6657467c4549b82e8d77f05432a6, entries=1973, sequenceid=21197, filesize=24.0 K 2022-03-06 01:24:50,017 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~171.20 KB/175310, heapSize ~317.63 KB/325256, currentSize=0 B/0 for 7534b3b1812808d2d3c16342fbf347a5 in 993ms, sequenceid=21197, compaction requested=true 2022-03-06 01:24:50,017 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 7534b3b1812808d2d3c16342fbf347a5/e in aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5. 2022-03-06 01:24:50,017 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/c5a168411a5f4dfeb9f36e26681c3050, hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/c55e42e7302d4c0ca206483e7855cf25, hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/0cee6657467c4549b82e8d77f05432a6] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/.tmp, totalSize=505.6 K 2022-03-06 01:24:50,094 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 7534b3b1812808d2d3c16342fbf347a5#e#compaction#216 average throughput is 59.15 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-06 01:24:50,167 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 7534b3b1812808d2d3c16342fbf347a5/e of 7534b3b1812808d2d3c16342fbf347a5 into 3b9f1d39056f47b3b966ba48b14b4f6c(size=498.9 K), total size for store is 498.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-06 01:24:50,167 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., storeName=7534b3b1812808d2d3c16342fbf347a5/e, priority=13, startTime=1646529890017; duration=0sec 2022-03-06 01:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=16127, hits=10495, hitRatio=65.08%, , cachingAccesses=10546, cachingHits=10456, cachingHitsRatio=99.15%, evictions=7854, evicted=0, evictedPerRun=0.0 2022-03-06 01:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 01:30:41,310 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1489 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:30:42,450 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 744 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:31:13,850 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646529673788 with entries=1289, filesize=562.81 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646530273840 2022-03-06 01:31:13,850 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646526070592 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646526070592 2022-03-06 01:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=16149, hits=10517, hitRatio=65.12%, , cachingAccesses=10568, cachingHits=10478, cachingHitsRatio=99.15%, evictions=7884, evicted=0, evictedPerRun=0.0 2022-03-06 01:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-06 01:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 01:31:53,640 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df. because 80a1ffe1dc2d8f258a6367d204dc64df/e has an old edit so flush to free WALs after random delay 257187 ms 2022-03-06 01:31:53,650 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1126ms No GCs detected 2022-03-06 01:32:23,074 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 713 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 01:32:33,332 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb. because ccdacf4b414cb8d1bc548efc77c663cb/e has an old edit so flush to free WALs after random delay 115504 ms 2022-03-06 01:33:53,876 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646529833809.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646530433864.meta 2022-03-06 01:33:53,876 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646529833809.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646529833809.meta 2022-03-06 01:34:28,837 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing ccdacf4b414cb8d1bc548efc77c663cb 1/1 column families, dataSize=294.02 KB heapSize=509.67 KB 2022-03-06 01:34:28,890 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=294.02 KB at sequenceid=14343 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/.tmp/e/f3680940eaef4e969bca94a9169889c3 2022-03-06 01:34:28,900 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/f3680940eaef4e969bca94a9169889c3, entries=2928, sequenceid=14343, filesize=30.5 K 2022-03-06 01:34:28,901 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~294.02 KB/301078, heapSize ~509.66 KB/521888, currentSize=0 B/0 for ccdacf4b414cb8d1bc548efc77c663cb in 64ms, sequenceid=14343, compaction requested=false 2022-03-06 01:36:10,828 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 80a1ffe1dc2d8f258a6367d204dc64df 1/1 column families, dataSize=305.47 KB heapSize=569.73 KB 2022-03-06 01:36:10,876 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=305.47 KB at sequenceid=26179 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/.tmp/e/40a2bc12af34414681c914e0bd28f445 2022-03-06 01:36:10,885 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/40a2bc12af34414681c914e0bd28f445, entries=3607, sequenceid=26179, filesize=40.6 K 2022-03-06 01:36:10,888 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~305.47 KB/312799, heapSize ~569.72 KB/583392, currentSize=0 B/0 for 80a1ffe1dc2d8f258a6367d204dc64df in 60ms, sequenceid=26179, compaction requested=false 2022-03-06 01:36:32,988 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. because 7d1b53259f1d8280a47f7704f98f2c9f/e has an old edit so flush to free WALs after random delay 238385 ms 2022-03-06 01:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=16177, hits=10545, hitRatio=65.19%, , cachingAccesses=10596, cachingHits=10506, cachingHitsRatio=99.15%, evictions=7914, evicted=0, evictedPerRun=0.0 2022-03-06 01:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 01:37:36,443 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 822 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 01:37:46,008 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 519 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 01:38:00,443 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 659 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 01:38:03,213 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2705 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 01:38:03,213 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2670 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 01:38:03,213 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2409 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 01:38:03,213 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2256 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 01:39:02,979 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391. because 0a764e51258807790269e46c55b99391/e has an old edit so flush to free WALs after random delay 114409 ms 2022-03-06 01:39:42,976 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e. because 4333b489291b1f77671b8f2e78779a1e/__status_family___ has an old edit so flush to free WALs after random delay 101840 ms 2022-03-06 01:40:31,372 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 7d1b53259f1d8280a47f7704f98f2c9f 1/1 column families, dataSize=680.57 KB heapSize=1.14 MB 2022-03-06 01:40:31,427 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=680.57 KB at sequenceid=24538 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/.tmp/e/0abafcfadef640728e1ebad54683a8f1 2022-03-06 01:40:31,437 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/0abafcfadef640728e1ebad54683a8f1, entries=6543, sequenceid=24538, filesize=64.8 K 2022-03-06 01:40:31,438 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~680.57 KB/696908, heapSize ~1.14 MB/1193888, currentSize=0 B/0 for 7d1b53259f1d8280a47f7704f98f2c9f in 66ms, sequenceid=24538, compaction requested=true 2022-03-06 01:40:31,438 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 7d1b53259f1d8280a47f7704f98f2c9f/e in aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. 2022-03-06 01:40:31,438 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/e03e0b809f6046ae90b8524a09f95cf8, hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/160d8782509444d1918349e6a944173b, hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/0abafcfadef640728e1ebad54683a8f1] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/.tmp, totalSize=1.7 M 2022-03-06 01:40:31,662 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 7d1b53259f1d8280a47f7704f98f2c9f#e#compaction#220 average throughput is 66.08 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-06 01:40:31,731 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 7d1b53259f1d8280a47f7704f98f2c9f/e of 7d1b53259f1d8280a47f7704f98f2c9f into 3f8ca2ba86544407a60384f23ce05f8e(size=1.7 M), total size for store is 1.7 M. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-06 01:40:31,731 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., storeName=7d1b53259f1d8280a47f7704f98f2c9f/e, priority=13, startTime=1646530831438; duration=0sec 2022-03-06 01:40:57,389 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 0a764e51258807790269e46c55b99391 1/1 column families, dataSize=920.27 KB heapSize=1.62 MB 2022-03-06 01:40:57,460 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=920.27 KB at sequenceid=50859 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/.tmp/e/93c5b7cee16a49c9b32fb93f5ff0b324 2022-03-06 01:40:57,468 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/93c5b7cee16a49c9b32fb93f5ff0b324, entries=9915, sequenceid=50859, filesize=91.0 K 2022-03-06 01:40:57,469 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~920.27 KB/942355, heapSize ~1.62 MB/1693920, currentSize=0 B/0 for 0a764e51258807790269e46c55b99391 in 80ms, sequenceid=50859, compaction requested=false 2022-03-06 01:41:13,906 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646530273840 with entries=1297, filesize=573.98 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646530873896 2022-03-06 01:41:24,816 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 4333b489291b1f77671b8f2e78779a1e 1/1 column families, dataSize=7.74 KB heapSize=12.36 KB 2022-03-06 01:41:24,913 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=7.74 KB at sequenceid=1463 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/.tmp/__status_family___/c6003268df6f4ea8afc3837d754d058f 2022-03-06 01:41:24,924 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/c6003268df6f4ea8afc3837d754d058f, entries=3, sequenceid=1463, filesize=5.1 K 2022-03-06 01:41:24,929 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~7.74 KB/7924, heapSize ~12.34 KB/12640, currentSize=0 B/0 for 4333b489291b1f77671b8f2e78779a1e in 113ms, sequenceid=1463, compaction requested=true 2022-03-06 01:41:24,930 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 4333b489291b1f77671b8f2e78779a1e/__status_family___ in __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e. 2022-03-06 01:41:24,930 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/ba985a3c1fe847f8871e04a223b58252, hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/e8c0224421f6444cba6d00b24f1e4a8a, hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/c6003268df6f4ea8afc3837d754d058f] into tmpdir=hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/.tmp, totalSize=15.4 K 2022-03-06 01:41:24,943 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 4333b489291b1f77671b8f2e78779a1e#__status_family___#compaction#223 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-06 01:41:25,007 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 4333b489291b1f77671b8f2e78779a1e/__status_family___ of 4333b489291b1f77671b8f2e78779a1e into 7115309d12c142b99c9ca34c5d411082(size=5.2 K), total size for store is 5.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-06 01:41:25,007 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=__test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., storeName=4333b489291b1f77671b8f2e78779a1e/__status_family___, priority=13, startTime=1646530884929; duration=0sec 2022-03-06 01:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=16433, hits=10567, hitRatio=64.30%, , cachingAccesses=10618, cachingHits=10528, cachingHitsRatio=99.15%, evictions=7944, evicted=0, evictedPerRun=0.0 2022-03-06 01:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 01:41:57,834 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 846 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 01:41:57,834 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 504 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 01:41:57,834 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 504 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 01:42:13,823 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3333 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 01:42:13,855 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2262 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 01:42:13,855 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1839 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 01:42:13,855 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1838 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 01:42:21,814 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 01:43:53,938 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646530433864.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646531033919.meta 2022-03-06 01:43:53,938 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646530433864.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646530433864.meta 2022-03-06 01:44:37,109 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1543 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 01:45:01,647 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 700 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 01:45:01,647 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 699 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 01:45:05,727 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3308 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 01:45:05,741 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2708 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 01:45:05,741 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2707 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 01:45:05,741 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1769 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 01:45:05,733 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2199ms No GCs detected 2022-03-06 01:45:13,722 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1816ms No GCs detected 2022-03-06 01:45:14,367 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3093 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 01:45:14,367 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2504 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 01:45:14,367 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1006 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 01:45:24,661 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1757 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 01:45:27,170 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 523 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 01:45:29,172 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1003ms No GCs detected 2022-03-06 01:45:46,530 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 884 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 01:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=16455, hits=10589, hitRatio=64.35%, , cachingAccesses=10640, cachingHits=10550, cachingHitsRatio=99.15%, evictions=7974, evicted=0, evictedPerRun=0.0 2022-03-06 01:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 01:47:17,880 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 682 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 01:47:33,700 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1261ms No GCs detected 2022-03-06 01:47:55,704 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 01:48:26,089 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3826ms No GCs detected 2022-03-06 01:48:28,428 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2344 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 01:48:28,428 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2341 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 01:48:28,440 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2249 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 01:48:30,256 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1395 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 01:48:30,256 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1395 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 01:50:31,859 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1952 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 01:50:31,859 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1631 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 01:50:31,945 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1560 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 01:50:31,945 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1522 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 01:50:31,945 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 833 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 01:51:16,292 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646530873896 with entries=1305, filesize=581.23 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646531476257 2022-03-06 01:51:16,292 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646526670627 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646526670627 2022-03-06 01:51:16,357 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646527270729 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646527270729 2022-03-06 01:51:16,393 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646527870772 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646527870772 2022-03-06 01:51:16,429 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646528472099 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646528472099 2022-03-06 01:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=16477, hits=10611, hitRatio=64.40%, , cachingAccesses=10662, cachingHits=10572, cachingHitsRatio=99.16%, evictions=8004, evicted=0, evictedPerRun=0.0 2022-03-06 01:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 01:52:03,380 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2562 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:52:04,131 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 751 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:52:04,744 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 616 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:52:09,196 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3024ms No GCs detected 2022-03-06 01:52:09,215 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2420 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:52:17,325 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1747 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:52:17,325 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1717 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:52:26,883 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 938 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:52:26,883 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 903 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:52:29,276 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646531033919.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646531549193.meta 2022-03-06 01:52:29,276 WARN [Close-WAL-Writer-158] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-06 01:52:29,276 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646531033919.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646531033919.meta 2022-03-06 01:52:29,276 WARN [Close-WAL-Writer-158] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-06 01:52:29,276 INFO [Close-WAL-Writer-158] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646531033919.meta 2022-03-06 01:52:29,336 INFO [Close-WAL-Writer-158] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646531033919.meta after 60ms 2022-03-06 01:52:33,339 WARN [Close-WAL-Writer-158] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646531033919.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646531033919.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-06 01:53:02,744 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 859 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:53:02,744 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 583 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:53:14,278 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1390 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:53:32,662 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 553 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:53:55,059 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 571 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:54:00,790 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2424ms No GCs detected 2022-03-06 01:54:02,149 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4220 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:54:02,149 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4220 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:54:02,149 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3190 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:54:02,149 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3187 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:54:02,149 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2778 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:54:02,149 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1345 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:54:02,149 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1114 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:54:02,149 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1010 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:54:03,505 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1355 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:54:03,505 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1311 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:54:03,505 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 643 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:54:11,114 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2601 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:54:15,457 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5701 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:54:16,346 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2459 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:54:16,346 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2450 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:54:16,346 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2367 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:54:16,346 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2211 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:54:16,346 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2114 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:54:22,491 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 936 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:54:29,118 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1179 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:54:29,119 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1002 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:56:16,193 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1983 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:56:16,193 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1963 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:56:16,193 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1123 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:56:16,196 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 715 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=283.85 KB, freeSize=306.92 MB, max=307.20 MB, blockCount=7, accesses=16499, hits=10633, hitRatio=64.45%, , cachingAccesses=10684, cachingHits=10594, cachingHitsRatio=99.16%, evictions=8034, evicted=0, evictedPerRun=0.0 2022-03-06 01:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 01:59:09,883 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1438 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:59:10,856 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1905ms No GCs detected 2022-03-06 01:59:25,641 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1967ms No GCs detected 2022-03-06 01:59:30,166 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: rg.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 01:59:31,240 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1306 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:59:31,248 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 836 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:59:31,248 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 815 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:59:31,248 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 663 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:59:31,248 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 534 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 01:59:31,394 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646531549193.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646531970164.meta 2022-03-06 01:59:31,394 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646531549193.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646531549193.meta 2022-03-06 01:59:31,394 WARN [Close-WAL-Writer-159] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-06 01:59:31,394 WARN [Close-WAL-Writer-159] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-06 01:59:31,395 INFO [Close-WAL-Writer-159] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646531549193.meta 2022-03-06 01:59:31,417 INFO [Close-WAL-Writer-159] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646531549193.meta after 22ms 2022-03-06 01:59:35,421 WARN [Close-WAL-Writer-159] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646531549193.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646531549193.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-06 01:59:53,568 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 718 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:00:06,228 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 652 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:00:14,177 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1065 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:00:14,910 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 683 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:00:14,910 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 643 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:00:23,612 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2080 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:00:23,612 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 707 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:00:23,612 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 624 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:00:27,381 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1826ms No GCs detected 2022-03-06 02:00:33,531 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 628 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:00:34,593 INFO [RpcServer.priority.RWQ.Fifo.read.handler=6,queue=1,port=16201] regionserver.RSRpcServices: Compacting aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5. 2022-03-06 02:00:34,594 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 7534b3b1812808d2d3c16342fbf347a5/e in aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5. 2022-03-06 02:00:34,594 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/3b9f1d39056f47b3b966ba48b14b4f6c] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/.tmp, totalSize=498.9 K 2022-03-06 02:00:34,608 INFO [RpcServer.priority.RWQ.Fifo.read.handler=5,queue=1,port=16201] regionserver.RSRpcServices: Compacting aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. 2022-03-06 02:00:34,608 INFO [regionserver/cvp328:16201-longCompactions-0] regionserver.HRegion: Starting compaction of 7d1b53259f1d8280a47f7704f98f2c9f/e in aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. 2022-03-06 02:00:34,609 INFO [regionserver/cvp328:16201-longCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/3f8ca2ba86544407a60384f23ce05f8e] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/.tmp, totalSize=1.7 M 2022-03-06 02:00:34,632 INFO [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=16201] regionserver.RSRpcServices: Compacting aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391. 2022-03-06 02:00:34,634 INFO [RpcServer.priority.RWQ.Fifo.read.handler=4,queue=1,port=16201] regionserver.RSRpcServices: Compacting aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb. 2022-03-06 02:00:34,640 INFO [RpcServer.priority.RWQ.Fifo.read.handler=8,queue=1,port=16201] regionserver.RSRpcServices: Compacting aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb. 2022-03-06 02:00:34,649 INFO [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=16201] regionserver.RSRpcServices: Compacting aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df. 2022-03-06 02:00:34,654 INFO [regionserver/cvp328:16201-longCompactions-0] retry.RetryInvocationHandler: doop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:433) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getBlockLocations over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 02:00:34,654 INFO [regionserver/cvp328:16201-shortCompactions-0] retry.RetryInvocationHandler: doop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:433) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getBlockLocations over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 02:00:34,654 WARN [regionserver/cvp328:16201-shortCompactions-0] retry.RetryInvocationHandler: A failover has occurred since the start of call #11659 ClientNamenodeProtocolTranslatorPB.getBlockLocations over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 2022-03-06 02:00:34,803 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 7534b3b1812808d2d3c16342fbf347a5#e#compaction#225 average throughput is 32.00 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2022-03-06 02:00:34,937 INFO [regionserver/cvp328:16201-longCompactions-0] throttle.PressureAwareThroughputController: 7d1b53259f1d8280a47f7704f98f2c9f#e#compaction#224 average throughput is 54.98 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-06 02:00:40,803 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4138 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:00:40,803 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3766 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:00:40,803 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2766ms No GCs detected 2022-03-06 02:00:44,363 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2059ms No GCs detected 2022-03-06 02:00:48,483 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2065ms No GCs detected 2022-03-06 02:00:50,389 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1908 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:00:50,390 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1895 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:00:53,656 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1296 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:00:53,656 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 829 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:01:00,976 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2939 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:01:00,976 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2936 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:01:02,128 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2462ms No GCs detected 2022-03-06 02:01:02,441 WARN [ResponseProcessor for block BP-1250263333-172.30.41.118-1646451070850:blk_1073747814_11528] hdfs.DataStreamer: Exception for BP-1250263333-172.30.41.118-1646451070850:blk_1073747814_11528 java.net.SocketTimeoutException: 25000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/172.30.41.118:35040 remote=/172.30.41.118:15010] at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118) at java.io.FilterInputStream.read(FilterInputStream.java:83) at java.io.FilterInputStream.read(FilterInputStream.java:83) at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:548) at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213) at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1086) 2022-03-06 02:01:02,442 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1473 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:01:02,445 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1473 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:01:02,442 WARN [DataStreamer for file /hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/.tmp/e/04a59ca994914f89ab8fa6dfcc32ec78 block BP-1250263333-172.30.41.118-1646451070850:blk_1073747814_11528] hdfs.DataStreamer: Error Recovery for BP-1250263333-172.30.41.118-1646451070850:blk_1073747814_11528 in pipeline [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]]: datanode 0(DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK]) is bad. 2022-03-06 02:01:04,490 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed major compaction of 1 (all) file(s) in 7534b3b1812808d2d3c16342fbf347a5/e of 7534b3b1812808d2d3c16342fbf347a5 into 04a59ca994914f89ab8fa6dfcc32ec78(size=498.9 K), total size for store is 498.9 K. This selection was in queue for 0sec, and took 29sec to execute. 2022-03-06 02:01:04,495 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction Request=regionName=aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., storeName=e, fileCount=1, fileSize=498.9 K (498.9 K), priority=1, time=1646532034594; duration=29sec 2022-03-06 02:01:04,495 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 0a764e51258807790269e46c55b99391/e in aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391. 2022-03-06 02:01:04,495 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/53326ff0062d4571b8205cad318872c6, hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/93c5b7cee16a49c9b32fb93f5ff0b324] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/.tmp, totalSize=2.4 M 2022-03-06 02:01:05,730 WARN [ResponseProcessor for block BP-1250263333-172.30.41.118-1646451070850:blk_1073747812_11526] hdfs.DataStreamer: Exception for BP-1250263333-172.30.41.118-1646451070850:blk_1073747812_11526 java.net.SocketTimeoutException: 25000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/172.30.41.118:35042 remote=/172.30.41.118:15010] at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118) at java.io.FilterInputStream.read(FilterInputStream.java:83) at java.io.FilterInputStream.read(FilterInputStream.java:83) at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:548) at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213) at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1086) 2022-03-06 02:01:05,730 WARN [DataStreamer for file /hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/.tmp/e/ef63d94635f14827ad8baf32533d856b block BP-1250263333-172.30.41.118-1646451070850:blk_1073747812_11526] hdfs.DataStreamer: Error Recovery for BP-1250263333-172.30.41.118-1646451070850:blk_1073747812_11526 in pipeline [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]]: datanode 0(DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK]) is bad. 2022-03-06 02:01:06,129 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 0a764e51258807790269e46c55b99391#e#compaction#226 average throughput is 44.59 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-06 02:01:06,185 WARN [regionserver/cvp328:16201-longCompactions-0] hdfs.DataStreamer: Slow waitForAckedSeqno took 31247ms (threshold=30000ms). File being written: /hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/.tmp/e/ef63d94635f14827ad8baf32533d856b, block: BP-1250263333-172.30.41.118-1646451070850:blk_1073747812_11541, Write pipeline datanodes: [DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]]. 2022-03-06 02:01:07,547 INFO [regionserver/cvp328:16201-longCompactions-0] regionserver.HStore: Completed major compaction of 1 (all) file(s) in 7d1b53259f1d8280a47f7704f98f2c9f/e of 7d1b53259f1d8280a47f7704f98f2c9f into ef63d94635f14827ad8baf32533d856b(size=1.7 M), total size for store is 1.7 M. This selection was in queue for 0sec, and took 32sec to execute. 2022-03-06 02:01:07,547 INFO [regionserver/cvp328:16201-longCompactions-0] regionserver.CompactSplit: Completed compaction Request=regionName=aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., storeName=e, fileCount=1, fileSize=1.7 M (1.7 M), priority=1, time=1646532034608; duration=32sec 2022-03-06 02:01:07,547 INFO [regionserver/cvp328:16201-longCompactions-0] regionserver.HRegion: Starting compaction of 9b121c22e18472616debd0996ee2a4cb/e in aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb. 2022-03-06 02:01:07,547 INFO [regionserver/cvp328:16201-longCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/3d547b9baa16448a9a317216f9f19370, hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/f0dc5dcf7ae0494dbb40102a5131bcc0] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/.tmp, totalSize=1.6 M 2022-03-06 02:01:09,707 INFO [regionserver/cvp328:16201-longCompactions-0] throttle.PressureAwareThroughputController: 9b121c22e18472616debd0996ee2a4cb#e#compaction#227 average throughput is 16.58 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-06 02:01:25,561 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3070ms No GCs detected 2022-03-06 02:01:25,587 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3613 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:01:25,863 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646531476257 with entries=1300, filesize=578.26 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646532082240 2022-03-06 02:01:27,926 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 545 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:01:31,158 WARN [ResponseProcessor for block BP-1250263333-172.30.41.118-1646451070850:blk_1073747821_11540] hdfs.DataStreamer: Exception for BP-1250263333-172.30.41.118-1646451070850:blk_1073747821_11540 java.net.SocketTimeoutException: 25000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/172.30.41.118:35060 remote=/172.30.41.118:15010] at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118) at java.io.FilterInputStream.read(FilterInputStream.java:83) at java.io.FilterInputStream.read(FilterInputStream.java:83) at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:548) at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213) at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1086) 2022-03-06 02:01:31,936 WARN [DataStreamer for file /hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/.tmp/e/7dbeb943396c45ca8019755a770a44e2 block BP-1250263333-172.30.41.118-1646451070850:blk_1073747821_11540] hdfs.DataStreamer: Error Recovery for BP-1250263333-172.30.41.118-1646451070850:blk_1073747821_11540 in pipeline [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]]: datanode 0(DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK]) is bad. 2022-03-06 02:01:31,938 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1154 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:01:34,072 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1141ms No GCs detected 2022-03-06 02:01:36,196 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4257 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:01:36,380 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4234 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:01:36,383 WARN [Close-WAL-Writer-155] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-06 02:01:36,401 INFO [Close-WAL-Writer-155] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646531476257 2022-03-06 02:01:36,374 WARN [ResponseProcessor for block BP-1250263333-172.30.41.118-1646451070850:blk_1073747822_11542] hdfs.DataStreamer: Exception for BP-1250263333-172.30.41.118-1646451070850:blk_1073747822_11542 java.net.SocketTimeoutException: 25000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/172.30.41.118:35066 remote=/172.30.41.118:15010] at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118) at java.io.FilterInputStream.read(FilterInputStream.java:83) at java.io.FilterInputStream.read(FilterInputStream.java:83) at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:548) at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213) at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1086) 2022-03-06 02:01:36,396 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1823ms No GCs detected 2022-03-06 02:01:36,406 WARN [DataStreamer for file /hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/.tmp/e/cc20666269bc451b821afb8deec88cbb block BP-1250263333-172.30.41.118-1646451070850:blk_1073747822_11542] hdfs.DataStreamer: Error Recovery for BP-1250263333-172.30.41.118-1646451070850:blk_1073747822_11542 in pipeline [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]]: datanode 0(DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK]) is bad. 2022-03-06 02:01:36,476 INFO [Close-WAL-Writer-155] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646531476257 after 75ms 2022-03-06 02:01:36,476 WARN [regionserver/cvp328:16201-shortCompactions-0] hdfs.DataStreamer: Slow waitForAckedSeqno took 30346ms (threshold=30000ms). File being written: /hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/.tmp/e/7dbeb943396c45ca8019755a770a44e2, block: BP-1250263333-172.30.41.118-1646451070850:blk_1073747821_11554, Write pipeline datanodes: null. 2022-03-06 02:01:36,730 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed major compaction of 2 (all) file(s) in 0a764e51258807790269e46c55b99391/e of 0a764e51258807790269e46c55b99391 into 7dbeb943396c45ca8019755a770a44e2(size=2.4 M), total size for store is 2.4 M. This selection was in queue for 29sec, and took 32sec to execute. 2022-03-06 02:01:36,731 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction Request=regionName=aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391., storeName=e, fileCount=2, fileSize=2.4 M (2.3 M, 91.0 K), priority=1, time=1646532034633; duration=32sec 2022-03-06 02:01:36,731 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of ccdacf4b414cb8d1bc548efc77c663cb/e in aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb. 2022-03-06 02:01:36,731 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/4e984961cc7b4f43a4b9dc57065e19e5, hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/f3680940eaef4e969bca94a9169889c3] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/.tmp, totalSize=787.5 K 2022-03-06 02:01:36,974 INFO [regionserver/cvp328:16201-longCompactions-0] regionserver.HStore: Completed major compaction of 2 (all) file(s) in 9b121c22e18472616debd0996ee2a4cb/e of 9b121c22e18472616debd0996ee2a4cb into cc20666269bc451b821afb8deec88cbb(size=1.6 M), total size for store is 1.6 M. This selection was in queue for 32sec, and took 29sec to execute. 2022-03-06 02:01:36,974 INFO [regionserver/cvp328:16201-longCompactions-0] regionserver.CompactSplit: Completed compaction Request=regionName=aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., storeName=e, fileCount=2, fileSize=1.6 M (1.5 M, 68.6 K), priority=1, time=1646532034634; duration=29sec 2022-03-06 02:01:36,975 INFO [regionserver/cvp328:16201-longCompactions-0] regionserver.HRegion: Starting compaction of 80a1ffe1dc2d8f258a6367d204dc64df/e in aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df. 2022-03-06 02:01:36,975 INFO [regionserver/cvp328:16201-longCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/2f56f68748df4891a59c60e076b28aab, hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/40a2bc12af34414681c914e0bd28f445] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/.tmp, totalSize=875.2 K 2022-03-06 02:01:37,074 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: ccdacf4b414cb8d1bc548efc77c663cb#e#compaction#228 average throughput is 34.70 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2022-03-06 02:01:37,153 INFO [regionserver/cvp328:16201-longCompactions-0] throttle.PressureAwareThroughputController: 80a1ffe1dc2d8f258a6367d204dc64df#e#compaction#229 average throughput is 63.14 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-06 02:01:40,503 INFO [Close-WAL-Writer-155] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646531476257 after 4102ms 2022-03-06 02:01:43,928 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1087ms No GCs detected 2022-03-06 02:01:44,389 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=4.19 MB, freeSize=303.01 MB, max=307.20 MB, blockCount=282, accesses=23577, hits=16635, hitRatio=70.56%, , cachingAccesses=16730, cachingHits=16331, cachingHitsRatio=97.62%, evictions=8062, evicted=0, evictedPerRun=0.0 2022-03-06 02:01:46,511 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1003ms No GCs detected 2022-03-06 02:01:47,185 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 675 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 02:01:53,209 INFO [Close-WAL-Writer-155] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=2 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646531476257 after 16808ms 2022-03-06 02:02:00,501 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1511ms No GCs detected 2022-03-06 02:02:00,633 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4131 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:02:00,633 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4037 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:02:02,103 WARN [ResponseProcessor for block BP-1250263333-172.30.41.118-1646451070850:blk_1073747831_11561] hdfs.DataStreamer: Exception for BP-1250263333-172.30.41.118-1646451070850:blk_1073747831_11561 java.net.SocketTimeoutException: 25000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/172.30.41.118:35086 remote=/172.30.41.118:15010] at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118) at java.io.FilterInputStream.read(FilterInputStream.java:83) at java.io.FilterInputStream.read(FilterInputStream.java:83) at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:548) at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213) at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1086) 2022-03-06 02:02:02,104 WARN [DataStreamer for file /hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/.tmp/e/f0445156166c4938b3ec6eada2467fed block BP-1250263333-172.30.41.118-1646451070850:blk_1073747831_11561] hdfs.DataStreamer: Error Recovery for BP-1250263333-172.30.41.118-1646451070850:blk_1073747831_11561 in pipeline [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]]: datanode 0(DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK]) is bad. 2022-03-06 02:02:03,064 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed major compaction of 2 (all) file(s) in ccdacf4b414cb8d1bc548efc77c663cb/e of ccdacf4b414cb8d1bc548efc77c663cb into f0445156166c4938b3ec6eada2467fed(size=792.0 K), total size for store is 792.0 K. This selection was in queue for 1mins, 2sec, and took 26sec to execute. 2022-03-06 02:02:03,064 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction Request=regionName=aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., storeName=e, fileCount=2, fileSize=787.5 K (757.0 K, 30.5 K), priority=1, time=1646532034640; duration=26sec 2022-03-06 02:02:05,193 WARN [ResponseProcessor for block BP-1250263333-172.30.41.118-1646451070850:blk_1073747832_11562] hdfs.DataStreamer: Exception for BP-1250263333-172.30.41.118-1646451070850:blk_1073747832_11562 java.net.SocketTimeoutException: 25000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/172.30.41.118:35088 remote=/172.30.41.118:15010] at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118) at java.io.FilterInputStream.read(FilterInputStream.java:83) at java.io.FilterInputStream.read(FilterInputStream.java:83) at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:548) at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213) at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1086) 2022-03-06 02:02:05,193 WARN [DataStreamer for file /hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/.tmp/e/e9c833cc814f4217b82557bcd6b76ed4 block BP-1250263333-172.30.41.118-1646451070850:blk_1073747832_11562] hdfs.DataStreamer: Error Recovery for BP-1250263333-172.30.41.118-1646451070850:blk_1073747832_11562 in pipeline [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]]: datanode 0(DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK]) is bad. 2022-03-06 02:02:05,328 INFO [regionserver/cvp328:16201-longCompactions-0] regionserver.HStore: Completed major compaction of 2 (all) file(s) in 80a1ffe1dc2d8f258a6367d204dc64df/e of 80a1ffe1dc2d8f258a6367d204dc64df into e9c833cc814f4217b82557bcd6b76ed4(size=870.7 K), total size for store is 870.7 K. This selection was in queue for 1mins, 2sec, and took 28sec to execute. 2022-03-06 02:02:05,328 INFO [regionserver/cvp328:16201-longCompactions-0] regionserver.CompactSplit: Completed compaction Request=regionName=aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., storeName=e, fileCount=2, fileSize=875.2 K (834.6 K, 40.6 K), priority=1, time=1646532034649; duration=28sec 2022-03-06 02:02:11,416 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2759 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:02:11,496 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 805 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:02:14,276 INFO [Close-WAL-Writer-155] retry.RetryInvocationHandler: apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.isFileClosed(ClientNamenodeProtocolServerSideTranslatorPB.java:1322) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.isFileClosed over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 02:02:14,393 INFO [Close-WAL-Writer-155] retry.RetryInvocationHandler: enodeProtocolServerSideTranslatorPB.isFileClosed(ClientNamenodeProtocolServerSideTranslatorPB.java:1322) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.isFileClosed over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 471ms. 2022-03-06 02:02:14,563 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 02:02:14,646 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 1 failover attempts. Trying to failover after sleeping for 416ms. 2022-03-06 02:02:14,864 WARN [Close-WAL-Writer-155] retry.RetryInvocationHandler: A failover has occurred since the start of call #11882 ClientNamenodeProtocolTranslatorPB.isFileClosed over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 2022-03-06 02:02:14,866 INFO [Close-WAL-Writer-155] retry.RetryInvocationHandler: nodeProtocolServerSideTranslatorPB.isFileClosed(ClientNamenodeProtocolServerSideTranslatorPB.java:1322) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.isFileClosed over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 2 failover attempts. Trying to failover after sleeping for 1719ms. 2022-03-06 02:02:15,073 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 2 failover attempts. Trying to failover after sleeping for 750ms. 2022-03-06 02:02:15,825 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: tNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 3 failover attempts. Trying to failover after sleeping for 3136ms. 2022-03-06 02:02:16,585 WARN [Close-WAL-Writer-155] retry.RetryInvocationHandler: A failover has occurred since the start of call #11882 ClientNamenodeProtocolTranslatorPB.isFileClosed over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 2022-03-06 02:02:16,586 INFO [Close-WAL-Writer-155] retry.RetryInvocationHandler: nodeProtocolServerSideTranslatorPB.isFileClosed(ClientNamenodeProtocolServerSideTranslatorPB.java:1322) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.isFileClosed over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 3 failover attempts. Trying to failover after sleeping for 2133ms. 2022-03-06 02:02:18,754 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1344 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:02:21,155 WARN [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: A failover has occurred since the start of call #11883 ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 2022-03-06 02:02:22,206 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2976ms No GCs detected 2022-03-06 02:02:22,359 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3604 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:02:25,675 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 686 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:02:29,896 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1988 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:02:29,896 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1988 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:02:29,896 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1955 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:02:29,896 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 767 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:02:29,896 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 762 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:02:33,559 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 591 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:02:34,966 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 775 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:02:46,306 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3961ms No GCs detected 2022-03-06 02:02:48,400 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 937 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:02:48,400 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 768 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:02:57,112 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1359ms No GCs detected 2022-03-06 02:03:07,737 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2667ms No GCs detected 2022-03-06 02:03:07,737 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5483 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:03:15,211 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1870ms No GCs detected 2022-03-06 02:03:19,461 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 975 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:03:21,163 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1226ms No GCs detected 2022-03-06 02:03:27,951 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 885 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:03:28,942 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 725 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:03:28,942 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 725 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:03:28,942 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 720 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:03:28,947 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 608 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:03:31,460 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2016ms No GCs detected 2022-03-06 02:03:34,829 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1249 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:03:34,831 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1185ms No GCs detected 2022-03-06 02:03:34,838 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 574 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:03:42,637 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: java.io.EOFException: End of File Exception between local host is: "cvp328.sjc.aristanetworks.com/172.30.41.118"; destination host is: "cvp328.sjc.aristanetworks.com":9001; : java.io.EOFException; For more details see: http://wiki.apache.org/hadoop/EOFException, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 4 failover attempts. Trying to failover after sleeping for 6437ms. 2022-03-06 02:03:42,637 INFO [Close-WAL-Writer-155] retry.RetryInvocationHandler: java.io.EOFException: End of File Exception between local host is: "cvp328.sjc.aristanetworks.com/172.30.41.118"; destination host is: "cvp328.sjc.aristanetworks.com":9001; : java.io.EOFException; For more details see: http://wiki.apache.org/hadoop/EOFException, while invoking ClientNamenodeProtocolTranslatorPB.isFileClosed over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 4 failover attempts. Trying to failover after sleeping for 3241ms. 2022-03-06 02:03:45,918 INFO [Close-WAL-Writer-155] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=3 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646531476257 after 129517ms 2022-03-06 02:03:45,918 WARN [Close-WAL-Writer-155] util.RecoverLeaseFSUtils: Cannot recoverLease after trying for 23000ms (hbase.lease.recovery.timeout); continuing, but may be DATALOSS!!!; attempt=3 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646531476257 after 129517ms 2022-03-06 02:03:49,741 WARN [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: A failover has occurred since the start of call #11883 ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 2022-03-06 02:03:49,743 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1887 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:03:52,985 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 528 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:03:59,749 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 911 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:03:59,750 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 908 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:03:59,750 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 822 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:04:07,025 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3257ms No GCs detected 2022-03-06 02:04:07,050 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4288 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:04:07,051 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4288 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:04:13,616 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 590 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:04:28,276 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 876 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:04:28,276 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 875 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:04:29,758 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1383 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:04:29,758 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 881 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:04:29,758 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 881 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:04:29,758 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 782 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:04:33,865 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1223 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:04:33,865 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1123 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:04:33,865 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1009 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:04:33,865 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 978 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:04:38,839 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5268 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:04:38,839 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4973 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:04:38,839 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4698 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:04:38,840 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3192 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:04:38,840 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3133 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:04:38,840 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1486 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:04:38,857 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 602 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:04:53,297 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 983 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:04:56,812 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1039 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:04:56,812 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 813 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:05:03,531 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3635 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:05:03,531 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3460 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:05:03,562 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3272ms No GCs detected 2022-03-06 02:05:37,380 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1801ms No GCs detected 2022-03-06 02:06:02,813 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 562 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:06:33,822 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1628 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:06:33,822 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1627 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:06:33,822 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1588 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:06:33,822 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 944 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:06:40,266 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3221ms No GCs detected 2022-03-06 02:06:40,327 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6505 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=300.27 KB, freeSize=306.91 MB, max=307.20 MB, blockCount=9, accesses=23599, hits=16655, hitRatio=70.58%, , cachingAccesses=16752, cachingHits=16351, cachingHitsRatio=97.61%, evictions=8092, evicted=0, evictedPerRun=0.0 2022-03-06 02:06:48,140 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 02:06:53,149 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 02:07:10,615 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646531970164.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646532430284.meta 2022-03-06 02:07:10,616 WARN [Close-WAL-Writer-160] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-06 02:07:10,616 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646531970164.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646531970164.meta 2022-03-06 02:07:10,616 WARN [Close-WAL-Writer-160] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-06 02:07:10,616 INFO [Close-WAL-Writer-160] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646531970164.meta 2022-03-06 02:07:10,723 INFO [Close-WAL-Writer-160] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646531970164.meta after 107ms 2022-03-06 02:07:14,726 WARN [Close-WAL-Writer-160] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646531970164.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646531970164.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-06 02:07:30,779 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 986 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:07:30,779 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 848 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:07:32,029 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1171 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:07:32,029 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1164 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:07:32,030 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1049 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:07:33,433 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1176 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:07:33,433 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 629 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:07:33,433 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 629 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:07:33,433 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 629 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:07:53,860 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 02:08:18,028 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 582 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:08:24,472 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 02:08:30,277 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4322 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:08:30,277 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3335 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:08:30,290 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 906 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:08:30,290 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 906 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:09:15,544 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1354 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:09:15,544 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1179 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:09:16,214 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 669 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:09:24,953 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 704 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:09:25,661 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1157 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:09:32,831 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 714 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:09:32,831 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 714 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:09:37,717 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 661 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:09:42,131 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1212 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:09:45,526 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4307 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:09:45,526 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4263 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:09:45,526 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3800 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:09:45,536 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2986ms No GCs detected 2022-03-06 02:09:49,838 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3011ms No GCs detected 2022-03-06 02:09:50,231 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4704 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:09:50,269 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4727 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:09:50,269 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4726 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:09:50,269 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4638 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:09:53,839 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1321ms No GCs detected 2022-03-06 02:09:53,875 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4034 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:09:53,875 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3605 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:09:53,875 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1848 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:10:25,025 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2260 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:10:28,772 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 02:10:31,138 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 737 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:10:31,138 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 703 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:10:31,835 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 695 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:10:33,644 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1309ms No GCs detected 2022-03-06 02:10:36,382 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3061 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:10:46,175 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1145 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:10:46,175 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 708 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:10:58,655 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 804 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:10:58,655 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 790 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:10:58,655 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 591 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:11:06,163 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 575 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:11:15,610 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 556 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:11:31,427 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646532082240 with entries=1315, filesize=574.95 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646532691388 2022-03-06 02:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=300.27 KB, freeSize=306.91 MB, max=307.20 MB, blockCount=9, accesses=23615, hits=16671, hitRatio=70.59%, , cachingAccesses=16768, cachingHits=16367, cachingHitsRatio=97.61%, evictions=8122, evicted=0, evictedPerRun=0.0 2022-03-06 02:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 02:11:59,569 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 583 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:12:06,692 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1037 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:12:09,181 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2177 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:12:18,244 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 917 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:12:18,244 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 815 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:12:28,724 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 834 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:12:28,724 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 808 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:12:28,724 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 808 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:12:28,724 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 680 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:14:18,339 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4218 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:14:18,339 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4064 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:14:18,339 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2823 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:14:18,339 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1112 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:14:18,339 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 909 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:14:34,938 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3499 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:14:34,938 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2773 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:14:34,938 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1888 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:14:34,938 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1885 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:14:34,938 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1743 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:14:34,939 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1504 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=300.27 KB, freeSize=306.91 MB, max=307.20 MB, blockCount=9, accesses=23637, hits=16693, hitRatio=70.62%, , cachingAccesses=16790, cachingHits=16389, cachingHitsRatio=97.61%, evictions=8152, evicted=0, evictedPerRun=0.0 2022-03-06 02:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 02:16:52,737 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1236 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:17:03,315 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 5449ms No GCs detected 2022-03-06 02:17:03,349 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4444 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:17:03,349 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4444 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:17:03,349 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4444 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:17:03,410 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4490 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:17:03,410 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4479 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:17:03,410 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1691 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:17:05,263 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1851 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:17:05,263 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1797 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:17:05,263 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1130 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:17:05,492 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 693 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:17:05,492 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 501 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:17:15,087 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646532430284.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646533033247.meta 2022-03-06 02:17:15,087 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646532430284.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646532430284.meta 2022-03-06 02:17:38,909 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1014ms No GCs detected 2022-03-06 02:17:40,386 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 609 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:17:42,604 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 810 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:17:42,604 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 804 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:17:59,670 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 821 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:17:59,671 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 717 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:18:02,762 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2443 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:18:02,813 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2431 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:18:02,813 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2370 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:18:02,813 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2091 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:18:56,987 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1098 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:18:58,366 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2351 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:18:59,542 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 680 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:19:02,399 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2855 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:19:02,399 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2382 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:19:02,399 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1982 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:19:02,399 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1979 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:19:02,399 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1201 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:19:22,980 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb. because 9b121c22e18472616debd0996ee2a4cb/e has an old edit so flush to free WALs after random delay 15749 ms 2022-03-06 02:19:38,729 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 9b121c22e18472616debd0996ee2a4cb 1/1 column families, dataSize=560.49 KB heapSize=1.01 MB 2022-03-06 02:19:38,770 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=560.49 KB at sequenceid=37811 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/.tmp/e/0ef30a00d80845e9be7f4d9e8f84ec7d 2022-03-06 02:19:38,782 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/0ef30a00d80845e9be7f4d9e8f84ec7d, entries=6495, sequenceid=37811, filesize=68.2 K 2022-03-06 02:19:38,783 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~560.49 KB/573944, heapSize ~1.01 MB/1061120, currentSize=0 B/0 for 9b121c22e18472616debd0996ee2a4cb in 54ms, sequenceid=37811, compaction requested=false 2022-03-06 02:21:35,510 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646532691388 with entries=1300, filesize=571.40 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646533295103 2022-03-06 02:21:35,519 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646529073585 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646529073585 2022-03-06 02:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=300.27 KB, freeSize=306.91 MB, max=307.20 MB, blockCount=9, accesses=23659, hits=16715, hitRatio=70.65%, , cachingAccesses=16812, cachingHits=16411, cachingHitsRatio=97.61%, evictions=8182, evicted=0, evictedPerRun=0.0 2022-03-06 02:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 02:22:53,607 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1945 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:22:56,205 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1960 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:22:56,205 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1822 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:22:59,135 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2912 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:22:59,135 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2666 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:22:59,136 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1255 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:22:59,136 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1243 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:22:59,136 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1243 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:22:59,136 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1006 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:23:24,405 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1630ms No GCs detected 2022-03-06 02:23:24,767 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 9106 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:23:24,767 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6100 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:23:24,767 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4689 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:23:26,129 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1361 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:23:27,319 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1383 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:23:27,319 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1302 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:23:27,319 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1296 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:23:27,319 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1189 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 02:23:27,319 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646533395066","responsesize":"78","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 11 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x06Kernel ","processingtimems":12253,"client":"172.31.0.219:43678","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":11} 2022-03-06 02:23:33,455 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1109ms No GCs detected 2022-03-06 02:23:40,656 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 02:23:40,661 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1556ms No GCs detected 2022-03-06 02:23:48,171 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 5206ms No GCs detected 2022-03-06 02:23:48,195 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-06 02:23:48,354 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-06 02:23:48,354 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-06 02:23:48,354 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-06 02:23:48,354 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.sync(AsyncProtobufLogWriter.java:148) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:394) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.consume(AsyncFSWAL.java:581) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-06 02:23:53,495 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1009ms No GCs detected 2022-03-06 02:23:56,833 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646533033247.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646533426350.meta 2022-03-06 02:23:56,833 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646533033247.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646533033247.meta 2022-03-06 02:23:56,834 WARN [Close-WAL-Writer-162] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-06 02:23:56,834 WARN [Close-WAL-Writer-162] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-06 02:23:56,834 INFO [Close-WAL-Writer-162] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646533033247.meta 2022-03-06 02:23:57,481 INFO [Close-WAL-Writer-162] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646533033247.meta after 647ms 2022-03-06 02:23:57,594 INFO [Log-Archiver-0] retry.RetryInvocationHandler: 040) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.rename(ClientNamenodeProtocolServerSideTranslatorPB.java:652) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.rename over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 02:23:57,594 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:568) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.addBlock over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 02:23:57,594 WARN [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: A failover has occurred since the start of call #11991 ClientNamenodeProtocolTranslatorPB.addBlock over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 2022-03-06 02:23:57,604 INFO [Log-Archiver-0] retry.RetryInvocationHandler: lPB.ClientNamenodeProtocolServerSideTranslatorPB.rename(ClientNamenodeProtocolServerSideTranslatorPB.java:652) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.rename over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 1 failover attempts. Trying to failover after sleeping for 654ms. 2022-03-06 02:23:57,604 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:568) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.addBlock over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 1 failover attempts. Trying to failover after sleeping for 818ms. 2022-03-06 02:23:58,265 INFO [Log-Archiver-0] retry.RetryInvocationHandler: lPB.ClientNamenodeProtocolServerSideTranslatorPB.rename(ClientNamenodeProtocolServerSideTranslatorPB.java:652) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.rename over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 2 failover attempts. Trying to failover after sleeping for 662ms. 2022-03-06 02:23:58,424 WARN [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: A failover has occurred since the start of call #11991 ClientNamenodeProtocolTranslatorPB.addBlock over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 2022-03-06 02:23:58,427 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: lientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:568) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.addBlock over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 2 failover attempts. Trying to failover after sleeping for 1212ms. 2022-03-06 02:23:59,697 WARN [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: A failover has occurred since the start of call #11991 ClientNamenodeProtocolTranslatorPB.addBlock over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 2022-03-06 02:24:06,470 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1572ms No GCs detected 2022-03-06 02:24:08,100 INFO [Close-WAL-Writer-162] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646533033247.meta after 11266ms 2022-03-06 02:24:09,513 WARN [Close-WAL-Writer-162] util.RecoverLeaseFSUtils: Failed invocation for /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646533033247.meta java.lang.reflect.InvocationTargetException at sun.reflect.GeneratedMethodAccessor176.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.isFileClosed(RecoverLeaseFSUtils.java:203) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:136) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646533033247.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.isFileClosed(FSDirStatAndListingOp.java:123) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.isFileClosed(FSNamesystem.java:3170) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.isFileClosed(NameNodeRpcServer.java:1204) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.isFileClosed(ClientNamenodeProtocolServerSideTranslatorPB.java:1322) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.isFileClosed(DFSClient.java:1707) at org.apache.hadoop.hdfs.DistributedFileSystem$45.doCall(DistributedFileSystem.java:2193) at org.apache.hadoop.hdfs.DistributedFileSystem$45.doCall(DistributedFileSystem.java:2190) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.isFileClosed(DistributedFileSystem.java:2208) ... 12 more Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646533033247.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.isFileClosed(FSDirStatAndListingOp.java:123) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.isFileClosed(FSNamesystem.java:3170) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.isFileClosed(NameNodeRpcServer.java:1204) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.isFileClosed(ClientNamenodeProtocolServerSideTranslatorPB.java:1322) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.isFileClosed(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.isFileClosed(ClientNamenodeProtocolTranslatorPB.java:1149) at sun.reflect.GeneratedMethodAccessor172.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.isFileClosed(Unknown Source) at sun.reflect.GeneratedMethodAccessor172.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.isFileClosed(Unknown Source) at sun.reflect.GeneratedMethodAccessor172.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.isFileClosed(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.isFileClosed(DFSClient.java:1705) ... 16 more 2022-03-06 02:24:09,788 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646533295103 with entries=268, filesize=117.27 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646533436833 2022-03-06 02:24:09,788 WARN [Close-WAL-Writer-158] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-06 02:24:09,789 WARN [Close-WAL-Writer-158] wal.AsyncProtobufLogWriter: normal close failed, try recover java.lang.IllegalStateException: should call flush first before calling close at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:510) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:514) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-06 02:24:09,789 INFO [Close-WAL-Writer-158] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646533295103 2022-03-06 02:24:09,790 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646533418606","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":31183,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-06 02:24:09,790 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646533422466","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":27323,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":1805,"class":"HRegionServer","multi.mutations":1} 2022-03-06 02:24:09,790 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646533422469","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":27320,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-06 02:24:09,790 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646533428355","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":21435,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 02:24:09,790 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646533428171","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":21619,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":1816,"class":"HRegionServer","multi.mutations":1} 2022-03-06 02:24:09,790 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646533435103","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":14687,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 02:24:09,790 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646533426355","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":23435,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 02:24:09,790 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646533420658","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":29131,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 02:24:09,790 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646533428196","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":21594,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":9,"class":"HRegionServer","multi.mutations":1} 2022-03-06 02:24:09,790 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646533428355","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":21435,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 02:24:09,790 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646533428359","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":21431,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":4,"class":"HRegionServer","multi.mutations":1} 2022-03-06 02:24:09,790 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646533435103","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":14687,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-06 02:24:09,790 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646533428196","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":21593,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 02:24:09,790 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646533435109","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":14681,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":7,"class":"HRegionServer","multi.mutations":1} 2022-03-06 02:24:09,790 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646533418605","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":31184,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 02:24:09,790 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646533426357","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":23432,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":2,"class":"HRegionServer","multi.mutations":1} 2022-03-06 02:24:09,882 INFO [Close-WAL-Writer-158] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646533295103 after 93ms 2022-03-06 02:24:10,779 INFO [Close-WAL-Writer-162] retry.RetryInvocationHandler: apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.isFileClosed(ClientNamenodeProtocolServerSideTranslatorPB.java:1322) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.isFileClosed over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 02:24:10,786 WARN [Close-WAL-Writer-162] util.RecoverLeaseFSUtils: Failed invocation for /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646533033247.meta java.lang.reflect.InvocationTargetException at sun.reflect.GeneratedMethodAccessor176.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.isFileClosed(RecoverLeaseFSUtils.java:203) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:136) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646533033247.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.isFileClosed(FSDirStatAndListingOp.java:123) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.isFileClosed(FSNamesystem.java:3170) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.isFileClosed(NameNodeRpcServer.java:1204) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.isFileClosed(ClientNamenodeProtocolServerSideTranslatorPB.java:1322) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.isFileClosed(DFSClient.java:1707) at org.apache.hadoop.hdfs.DistributedFileSystem$45.doCall(DistributedFileSystem.java:2193) at org.apache.hadoop.hdfs.DistributedFileSystem$45.doCall(DistributedFileSystem.java:2190) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.isFileClosed(DistributedFileSystem.java:2208) ... 12 more Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646533033247.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.isFileClosed(FSDirStatAndListingOp.java:123) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.isFileClosed(FSNamesystem.java:3170) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.isFileClosed(NameNodeRpcServer.java:1204) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.isFileClosed(ClientNamenodeProtocolServerSideTranslatorPB.java:1322) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.isFileClosed(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.isFileClosed(ClientNamenodeProtocolTranslatorPB.java:1149) at sun.reflect.GeneratedMethodAccessor172.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.isFileClosed(Unknown Source) at sun.reflect.GeneratedMethodAccessor172.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.isFileClosed(Unknown Source) at sun.reflect.GeneratedMethodAccessor172.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.isFileClosed(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.isFileClosed(DFSClient.java:1705) ... 16 more 2022-03-06 02:24:11,789 WARN [Close-WAL-Writer-162] util.RecoverLeaseFSUtils: Failed invocation for /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646533033247.meta java.lang.reflect.InvocationTargetException at sun.reflect.GeneratedMethodAccessor176.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.isFileClosed(RecoverLeaseFSUtils.java:203) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:136) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646533033247.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.isFileClosed(FSDirStatAndListingOp.java:123) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.isFileClosed(FSNamesystem.java:3170) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.isFileClosed(NameNodeRpcServer.java:1204) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.isFileClosed(ClientNamenodeProtocolServerSideTranslatorPB.java:1322) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.isFileClosed(DFSClient.java:1707) at org.apache.hadoop.hdfs.DistributedFileSystem$45.doCall(DistributedFileSystem.java:2193) at org.apache.hadoop.hdfs.DistributedFileSystem$45.doCall(DistributedFileSystem.java:2190) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.isFileClosed(DistributedFileSystem.java:2208) ... 12 more Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646533033247.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.isFileClosed(FSDirStatAndListingOp.java:123) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.isFileClosed(FSNamesystem.java:3170) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.isFileClosed(NameNodeRpcServer.java:1204) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.isFileClosed(ClientNamenodeProtocolServerSideTranslatorPB.java:1322) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.isFileClosed(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.isFileClosed(ClientNamenodeProtocolTranslatorPB.java:1149) at sun.reflect.GeneratedMethodAccessor172.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.isFileClosed(Unknown Source) at sun.reflect.GeneratedMethodAccessor172.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.isFileClosed(Unknown Source) at sun.reflect.GeneratedMethodAccessor172.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.isFileClosed(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.isFileClosed(DFSClient.java:1705) ... 16 more 2022-03-06 02:24:12,799 WARN [Close-WAL-Writer-162] util.RecoverLeaseFSUtils: Failed invocation for /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646533033247.meta java.lang.reflect.InvocationTargetException at sun.reflect.GeneratedMethodAccessor176.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.isFileClosed(RecoverLeaseFSUtils.java:203) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:136) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646533033247.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.isFileClosed(FSDirStatAndListingOp.java:123) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.isFileClosed(FSNamesystem.java:3170) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.isFileClosed(NameNodeRpcServer.java:1204) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.isFileClosed(ClientNamenodeProtocolServerSideTranslatorPB.java:1322) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.isFileClosed(DFSClient.java:1707) at org.apache.hadoop.hdfs.DistributedFileSystem$45.doCall(DistributedFileSystem.java:2193) at org.apache.hadoop.hdfs.DistributedFileSystem$45.doCall(DistributedFileSystem.java:2190) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.isFileClosed(DistributedFileSystem.java:2208) ... 12 more Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646533033247.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.isFileClosed(FSDirStatAndListingOp.java:123) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.isFileClosed(FSNamesystem.java:3170) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.isFileClosed(NameNodeRpcServer.java:1204) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.isFileClosed(ClientNamenodeProtocolServerSideTranslatorPB.java:1322) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.isFileClosed(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.isFileClosed(ClientNamenodeProtocolTranslatorPB.java:1149) at sun.reflect.GeneratedMethodAccessor172.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.isFileClosed(Unknown Source) at sun.reflect.GeneratedMethodAccessor172.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.isFileClosed(Unknown Source) at sun.reflect.GeneratedMethodAccessor172.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.isFileClosed(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.isFileClosed(DFSClient.java:1705) ... 16 more 2022-03-06 02:24:13,807 WARN [Close-WAL-Writer-162] util.RecoverLeaseFSUtils: Failed invocation for /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646533033247.meta java.lang.reflect.InvocationTargetException at sun.reflect.GeneratedMethodAccessor176.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.isFileClosed(RecoverLeaseFSUtils.java:203) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:136) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646533033247.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.isFileClosed(FSDirStatAndListingOp.java:123) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.isFileClosed(FSNamesystem.java:3170) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.isFileClosed(NameNodeRpcServer.java:1204) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.isFileClosed(ClientNamenodeProtocolServerSideTranslatorPB.java:1322) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.isFileClosed(DFSClient.java:1707) at org.apache.hadoop.hdfs.DistributedFileSystem$45.doCall(DistributedFileSystem.java:2193) at org.apache.hadoop.hdfs.DistributedFileSystem$45.doCall(DistributedFileSystem.java:2190) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.isFileClosed(DistributedFileSystem.java:2208) ... 12 more Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646533033247.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.isFileClosed(FSDirStatAndListingOp.java:123) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.isFileClosed(FSNamesystem.java:3170) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.isFileClosed(NameNodeRpcServer.java:1204) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.isFileClosed(ClientNamenodeProtocolServerSideTranslatorPB.java:1322) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.isFileClosed(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.isFileClosed(ClientNamenodeProtocolTranslatorPB.java:1149) at sun.reflect.GeneratedMethodAccessor172.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.isFileClosed(Unknown Source) at sun.reflect.GeneratedMethodAccessor172.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.isFileClosed(Unknown Source) at sun.reflect.GeneratedMethodAccessor172.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.isFileClosed(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.isFileClosed(DFSClient.java:1705) ... 16 more 2022-03-06 02:24:13,989 INFO [Close-WAL-Writer-158] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646533295103 after 4200ms 2022-03-06 02:24:14,809 WARN [Close-WAL-Writer-162] util.RecoverLeaseFSUtils: Failed invocation for /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646533033247.meta java.lang.reflect.InvocationTargetException at sun.reflect.GeneratedMethodAccessor176.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.isFileClosed(RecoverLeaseFSUtils.java:203) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:136) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646533033247.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.isFileClosed(FSDirStatAndListingOp.java:123) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.isFileClosed(FSNamesystem.java:3170) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.isFileClosed(NameNodeRpcServer.java:1204) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.isFileClosed(ClientNamenodeProtocolServerSideTranslatorPB.java:1322) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.isFileClosed(DFSClient.java:1707) at org.apache.hadoop.hdfs.DistributedFileSystem$45.doCall(DistributedFileSystem.java:2193) at org.apache.hadoop.hdfs.DistributedFileSystem$45.doCall(DistributedFileSystem.java:2190) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.isFileClosed(DistributedFileSystem.java:2208) ... 12 more Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646533033247.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.isFileClosed(FSDirStatAndListingOp.java:123) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.isFileClosed(FSNamesystem.java:3170) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.isFileClosed(NameNodeRpcServer.java:1204) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.isFileClosed(ClientNamenodeProtocolServerSideTranslatorPB.java:1322) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.isFileClosed(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.isFileClosed(ClientNamenodeProtocolTranslatorPB.java:1149) at sun.reflect.GeneratedMethodAccessor172.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.isFileClosed(Unknown Source) at sun.reflect.GeneratedMethodAccessor172.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.isFileClosed(Unknown Source) at sun.reflect.GeneratedMethodAccessor172.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.isFileClosed(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.isFileClosed(DFSClient.java:1705) ... 16 more 2022-03-06 02:24:18,331 WARN [Close-WAL-Writer-162] util.RecoverLeaseFSUtils: Failed invocation for /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646533033247.meta java.lang.reflect.InvocationTargetException at sun.reflect.GeneratedMethodAccessor176.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.isFileClosed(RecoverLeaseFSUtils.java:203) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:136) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646533033247.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.isFileClosed(FSDirStatAndListingOp.java:123) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.isFileClosed(FSNamesystem.java:3170) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.isFileClosed(NameNodeRpcServer.java:1204) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.isFileClosed(ClientNamenodeProtocolServerSideTranslatorPB.java:1322) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.isFileClosed(DFSClient.java:1707) at org.apache.hadoop.hdfs.DistributedFileSystem$45.doCall(DistributedFileSystem.java:2193) at org.apache.hadoop.hdfs.DistributedFileSystem$45.doCall(DistributedFileSystem.java:2190) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.isFileClosed(DistributedFileSystem.java:2208) ... 12 more Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646533033247.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.isFileClosed(FSDirStatAndListingOp.java:123) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.isFileClosed(FSNamesystem.java:3170) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.isFileClosed(NameNodeRpcServer.java:1204) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.isFileClosed(ClientNamenodeProtocolServerSideTranslatorPB.java:1322) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.isFileClosed(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.isFileClosed(ClientNamenodeProtocolTranslatorPB.java:1149) at sun.reflect.GeneratedMethodAccessor172.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.isFileClosed(Unknown Source) at sun.reflect.GeneratedMethodAccessor172.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.isFileClosed(Unknown Source) at sun.reflect.GeneratedMethodAccessor172.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.isFileClosed(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.isFileClosed(DFSClient.java:1705) ... 16 more 2022-03-06 02:24:19,333 INFO [Close-WAL-Writer-162] retry.RetryInvocationHandler: apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.isFileClosed(ClientNamenodeProtocolServerSideTranslatorPB.java:1322) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.isFileClosed over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 02:24:19,335 WARN [Close-WAL-Writer-162] util.RecoverLeaseFSUtils: Failed invocation for /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646533033247.meta java.lang.reflect.InvocationTargetException at sun.reflect.GeneratedMethodAccessor176.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.isFileClosed(RecoverLeaseFSUtils.java:203) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:136) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646533033247.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.isFileClosed(FSDirStatAndListingOp.java:123) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.isFileClosed(FSNamesystem.java:3170) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.isFileClosed(NameNodeRpcServer.java:1204) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.isFileClosed(ClientNamenodeProtocolServerSideTranslatorPB.java:1322) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.isFileClosed(DFSClient.java:1707) at org.apache.hadoop.hdfs.DistributedFileSystem$45.doCall(DistributedFileSystem.java:2193) at org.apache.hadoop.hdfs.DistributedFileSystem$45.doCall(DistributedFileSystem.java:2190) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.isFileClosed(DistributedFileSystem.java:2208) ... 12 more Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646533033247.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.isFileClosed(FSDirStatAndListingOp.java:123) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.isFileClosed(FSNamesystem.java:3170) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.isFileClosed(NameNodeRpcServer.java:1204) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.isFileClosed(ClientNamenodeProtocolServerSideTranslatorPB.java:1322) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.isFileClosed(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.isFileClosed(ClientNamenodeProtocolTranslatorPB.java:1149) at sun.reflect.GeneratedMethodAccessor172.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.isFileClosed(Unknown Source) at sun.reflect.GeneratedMethodAccessor172.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.isFileClosed(Unknown Source) at sun.reflect.GeneratedMethodAccessor172.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.isFileClosed(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.isFileClosed(DFSClient.java:1705) ... 16 more 2022-03-06 02:24:19,336 WARN [Close-WAL-Writer-162] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646533033247.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646533033247.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-06 02:24:31,061 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2240 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:24:31,061 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2180 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:24:36,491 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 872 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:25:01,538 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1592 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:25:01,538 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1474 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:25:02,032 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1611 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:25:02,032 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1611 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:25:02,032 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1265 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:25:02,032 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1242 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:25:03,407 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5. because 7534b3b1812808d2d3c16342fbf347a5/e has an old edit so flush to free WALs after random delay 55870 ms 2022-03-06 02:25:04,021 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1851 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:25:04,022 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1792 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:25:04,093 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1332 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:25:04,093 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1332 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:25:10,975 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1078ms No GCs detected 2022-03-06 02:25:12,905 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1734 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:25:12,905 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1624 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:25:12,905 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1124 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:25:15,432 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1030ms No GCs detected 2022-03-06 02:25:16,902 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3898 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:25:16,963 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1526 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:25:16,963 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1522 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:25:24,908 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 726 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:25:47,044 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 02:25:59,279 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 7534b3b1812808d2d3c16342fbf347a5 1/1 column families, dataSize=171.60 KB heapSize=318.01 KB 2022-03-06 02:25:59,351 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=171.60 KB at sequenceid=22161 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/.tmp/e/88224c4284844dc1800f1cfe6209bcdd 2022-03-06 02:25:59,359 INFO [MemStoreFlusher.0] retry.RetryInvocationHandler: doop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:433) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getBlockLocations over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 02:25:59,409 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/88224c4284844dc1800f1cfe6209bcdd, entries=1972, sequenceid=22161, filesize=24.4 K 2022-03-06 02:25:59,411 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~171.60 KB/175715, heapSize ~317.99 KB/325624, currentSize=0 B/0 for 7534b3b1812808d2d3c16342fbf347a5 in 132ms, sequenceid=22161, compaction requested=false 2022-03-06 02:26:12,653 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 540 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=300.27 KB, freeSize=306.91 MB, max=307.20 MB, blockCount=9, accesses=23681, hits=16737, hitRatio=70.68%, , cachingAccesses=16834, cachingHits=16433, cachingHitsRatio=97.62%, evictions=8212, evicted=0, evictedPerRun=0.0 2022-03-06 02:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 02:27:11,272 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4292 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:27:11,272 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3071 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:27:11,272 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2807 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:27:11,272 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1465 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:27:37,046 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1242 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:27:40,210 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1966 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:27:40,210 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1758 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:27:42,701 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1244ms No GCs detected 2022-03-06 02:27:42,995 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1699 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:27:43,405 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 644 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:27:43,405 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 589 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:27:45,767 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2362 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:27:45,767 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 904 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:27:52,795 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1546ms No GCs detected 2022-03-06 02:27:57,426 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4131ms No GCs detected 2022-03-06 02:27:57,436 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-06 02:28:01,005 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646533436833 with entries=537, filesize=236.46 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646533677670 2022-03-06 02:28:01,005 WARN [Close-WAL-Writer-159] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-06 02:28:01,007 WARN [Close-WAL-Writer-159] wal.AsyncProtobufLogWriter: normal close failed, try recover java.lang.IllegalStateException: should call flush first before calling close at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:510) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:514) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-06 02:28:01,007 INFO [Close-WAL-Writer-159] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646533436833 2022-03-06 02:28:01,005 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646529673788 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646529673788 2022-03-06 02:28:01,014 INFO [Close-WAL-Writer-159] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646533436833 after 7ms 2022-03-06 02:28:05,015 INFO [Close-WAL-Writer-159] util.RecoverLeaseFSUtils: Recovered lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646533436833 after 4008ms 2022-03-06 02:28:42,740 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1468 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:28:42,740 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1444 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:28:42,740 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1038 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:28:53,024 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 725 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:29:15,666 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2828 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:29:15,666 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1458 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:29:15,666 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1457 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:29:41,757 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 539 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:30:37,875 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 509 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=300.27 KB, freeSize=306.91 MB, max=307.20 MB, blockCount=9, accesses=23703, hits=16759, hitRatio=70.70%, , cachingAccesses=16856, cachingHits=16455, cachingHitsRatio=97.62%, evictions=8241, evicted=0, evictedPerRun=0.0 2022-03-06 02:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-06 02:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 02:33:16,174 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 654 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:33:39,133 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 862 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:33:39,133 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 658 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:33:51,881 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646533426350.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646534031009.meta 2022-03-06 02:33:51,881 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646533426350.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646533426350.meta 2022-03-06 02:34:01,726 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 791 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:34:42,976 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb. because ccdacf4b414cb8d1bc548efc77c663cb/e has an old edit so flush to free WALs after random delay 141035 ms 2022-03-06 02:34:46,566 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1171 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:34:46,566 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1090 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:35:35,018 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 834 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:35:36,158 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 562 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:35:39,525 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2506 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:35:39,529 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2135ms No GCs detected 2022-03-06 02:35:44,919 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3123 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:35:44,919 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3121 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:35:44,919 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3120 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:35:46,043 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2990 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:35:47,636 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1590 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:35:47,636 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1587 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:35:47,636 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1587 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:35:49,973 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: java.net.ConnectException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp328.sjc.aristanetworks.com:9001 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 02:35:54,673 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1328ms No GCs detected 2022-03-06 02:35:55,927 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1345 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:35:59,145 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3292 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:36:00,575 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1430 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:36:00,575 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 507 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:36:00,575 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 505 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:36:01,689 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1114 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:36:01,690 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1112 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:36:01,690 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1029 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:36:01,695 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1021 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:36:01,695 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 755 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:36:01,695 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 732 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:36:22,988 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df. because 80a1ffe1dc2d8f258a6367d204dc64df/e has an old edit so flush to free WALs after random delay 288294 ms 2022-03-06 02:36:32,165 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5516 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:36:32,165 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4667 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:36:32,165 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4667 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:36:32,165 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3396 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:36:42,218 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 674 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:36:43,352 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1135 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:36:44,384 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=300.27 KB, freeSize=306.91 MB, max=307.20 MB, blockCount=9, accesses=23731, hits=16787, hitRatio=70.74%, , cachingAccesses=16884, cachingHits=16483, cachingHitsRatio=97.62%, evictions=8271, evicted=0, evictedPerRun=0.0 2022-03-06 02:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 02:37:04,011 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing ccdacf4b414cb8d1bc548efc77c663cb 1/1 column families, dataSize=292.58 KB heapSize=505.34 KB 2022-03-06 02:37:04,131 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=292.58 KB at sequenceid=14965 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/.tmp/e/b1e690a811a64b30857392d00d21b91e 2022-03-06 02:37:04,137 INFO [MemStoreFlusher.0] retry.RetryInvocationHandler: doop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:433) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getBlockLocations over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 02:37:04,143 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/b1e690a811a64b30857392d00d21b91e, entries=2888, sequenceid=14965, filesize=30.5 K 2022-03-06 02:37:04,144 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~292.58 KB/299601, heapSize ~505.32 KB/517448, currentSize=0 B/0 for ccdacf4b414cb8d1bc548efc77c663cb in 133ms, sequenceid=14965, compaction requested=false 2022-03-06 02:38:03,404 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646533677670 with entries=1319, filesize=583.54 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646534282210 2022-03-06 02:38:23,074 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 711 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:38:37,831 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 815 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:39:10,392 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 596 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:39:12,856 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 876 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:39:12,856 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 722 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:39:45,084 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 658 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:39:45,084 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 658 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:40:12,544 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 835 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:40:22,770 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2872 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:40:22,770 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1030 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:40:24,789 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 02:40:26,500 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4208 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:40:26,501 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2273 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:40:26,524 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 599ms. 2022-03-06 02:40:27,260 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 656 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:40:30,753 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 796 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:40:32,421 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1980 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:40:32,422 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1636 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:40:32,422 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1499 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:40:32,995 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. because 7d1b53259f1d8280a47f7704f98f2c9f/e has an old edit so flush to free WALs after random delay 155767 ms 2022-03-06 02:40:46,490 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1287ms No GCs detected 2022-03-06 02:40:54,228 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 808 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:40:54,228 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 803 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:41:02,987 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391. because 0a764e51258807790269e46c55b99391/e has an old edit so flush to free WALs after random delay 127733 ms 2022-03-06 02:41:10,023 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2465 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:41:10,024 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1555 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:41:10,024 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1530 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:41:10,024 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1204 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:41:11,283 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 80a1ffe1dc2d8f258a6367d204dc64df 1/1 column families, dataSize=313.17 KB heapSize=584.83 KB 2022-03-06 02:41:11,557 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=313.17 KB at sequenceid=27421 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/.tmp/e/9a9d9d397a6e47968a8792a1d9f97fb3 2022-03-06 02:41:11,602 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/9a9d9d397a6e47968a8792a1d9f97fb3, entries=3708, sequenceid=27421, filesize=41.5 K 2022-03-06 02:41:11,603 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~313.17 KB/320681, heapSize ~584.81 KB/598848, currentSize=0 B/0 for 80a1ffe1dc2d8f258a6367d204dc64df in 320ms, sequenceid=27421, compaction requested=false 2022-03-06 02:41:25,273 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 711 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:41:27,990 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1904 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:41:32,742 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 565 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:41:32,742 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 565 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:41:34,812 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1432 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:41:34,812 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 628 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:41:45,201 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2078ms No GCs detected 2022-03-06 02:41:45,211 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e. because 4333b489291b1f77671b8f2e78779a1e/__status_family___ has an old edit so flush to free WALs after random delay 147867 ms 2022-03-06 02:41:46,841 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1139ms No GCs detected 2022-03-06 02:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 02:41:46,873 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1661 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:41:52,727 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 7505 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:41:52,762 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5909 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:41:46,842 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=300.27 KB, freeSize=306.91 MB, max=307.20 MB, blockCount=9, accesses=23753, hits=16809, hitRatio=70.77%, , cachingAccesses=16906, cachingHits=16505, cachingHitsRatio=97.63%, evictions=8301, evicted=0, evictedPerRun=0.0 2022-03-06 02:41:58,731 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 02:42:04,544 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1814 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:42:04,953 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2223 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:42:04,953 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1854 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:42:04,953 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1553 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:42:04,953 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 822 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:42:08,315 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1508ms No GCs detected 2022-03-06 02:42:08,494 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2077 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:42:13,316 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2154 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:42:13,316 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2014 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:42:13,316 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1613 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:42:25,058 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 888 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:42:25,059 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 559 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:42:26,668 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 867 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:42:26,668 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 738 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:42:26,668 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 710 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:43:00,835 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1424ms No GCs detected 2022-03-06 02:43:01,424 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3304 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:43:01,424 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2509 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:43:01,424 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1046 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:43:01,429 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 597 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:43:01,429 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 576 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:43:08,762 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 7d1b53259f1d8280a47f7704f98f2c9f 1/1 column families, dataSize=660.46 KB heapSize=1.10 MB 2022-03-06 02:43:08,765 INFO [MemStoreFlusher.1] retry.RetryInvocationHandler: rg.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 02:43:10,720 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 0a764e51258807790269e46c55b99391 1/1 column families, dataSize=929.22 KB heapSize=1.63 MB 2022-03-06 02:43:12,563 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=660.46 KB at sequenceid=25652 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/.tmp/e/06b95ff5019b4759bfcd2ae50d6bf3e5 2022-03-06 02:43:12,594 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/06b95ff5019b4759bfcd2ae50d6bf3e5, entries=6349, sequenceid=25652, filesize=63.5 K 2022-03-06 02:43:12,595 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~660.46 KB/676306, heapSize ~1.10 MB/1158576, currentSize=859 B/859 for 7d1b53259f1d8280a47f7704f98f2c9f in 3833ms, sequenceid=25652, compaction requested=false 2022-03-06 02:43:12,646 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=929.22 KB at sequenceid=53229 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/.tmp/e/c86941e1554349f6b32f37ca3c8dd508 2022-03-06 02:43:12,670 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/c86941e1554349f6b32f37ca3c8dd508, entries=10007, sequenceid=53229, filesize=92.1 K 2022-03-06 02:43:12,672 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~929.22 KB/951521, heapSize ~1.63 MB/1710088, currentSize=1.07 KB/1097 for 0a764e51258807790269e46c55b99391 in 1952ms, sequenceid=53229, compaction requested=false 2022-03-06 02:43:22,628 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1101 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:43:31,512 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 751 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:43:33,159 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1029 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:43:33,932 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 752 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:43:33,932 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 752 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:43:33,932 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 749 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:43:46,882 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 816 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:43:56,951 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646534031009.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646534635195.meta 2022-03-06 02:43:56,951 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646534031009.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646534031009.meta 2022-03-06 02:44:09,092 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 648 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:44:13,078 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 4333b489291b1f77671b8f2e78779a1e 1/1 column families, dataSize=7.00 KB heapSize=11.20 KB 2022-03-06 02:44:13,124 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=7.00 KB at sequenceid=1523 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/.tmp/__status_family___/dbc8d0a0e30941f08abf4eb28c7a90da 2022-03-06 02:44:13,136 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/dbc8d0a0e30941f08abf4eb28c7a90da, entries=3, sequenceid=1523, filesize=5.1 K 2022-03-06 02:44:13,138 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~7.00 KB/7164, heapSize ~11.19 KB/11456, currentSize=0 B/0 for 4333b489291b1f77671b8f2e78779a1e in 60ms, sequenceid=1523, compaction requested=false 2022-03-06 02:44:31,468 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 903 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:44:31,468 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 671 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:44:32,707 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 512 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:44:32,707 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 512 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:44:34,199 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 02:45:11,735 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 546 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:45:23,056 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 718 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:45:29,231 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 5460ms No GCs detected 2022-03-06 02:45:32,337 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2363 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:45:32,337 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2167 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:45:32,337 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2167 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:45:32,337 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2164 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:45:32,337 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 899 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:45:32,337 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 897 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:45:38,716 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 503 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:45:40,395 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 609 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:46:22,209 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5030 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:46:22,209 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4813 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:46:22,209 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 763 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:46:28,426 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3207 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:46:28,426 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2508 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:46:28,426 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 870 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:46:31,347 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2533 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:46:31,347 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2394 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:46:31,347 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2290 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:46:31,347 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1425 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:46:31,347 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 934 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:46:31,347 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 662 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:46:33,420 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1663 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:46:33,420 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1262 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:46:33,420 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1206 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:46:34,450 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 915 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:46:36,845 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2703 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:46:36,845 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2390 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:46:36,845 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1199 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:46:38,274 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 915 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=300.27 KB, freeSize=306.91 MB, max=307.20 MB, blockCount=9, accesses=23775, hits=16831, hitRatio=70.79%, , cachingAccesses=16928, cachingHits=16527, cachingHitsRatio=97.63%, evictions=8331, evicted=0, evictedPerRun=0.0 2022-03-06 02:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 02:47:22,867 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1354 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:47:22,867 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 552 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:47:32,953 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 676 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:47:35,444 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 808 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:47:58,406 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 593 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:48:10,269 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646534282210 with entries=1320, filesize=582.94 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646534889273 2022-03-06 02:48:10,269 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646530273840 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646530273840 2022-03-06 02:48:10,327 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646530873896 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646530873896 2022-03-06 02:48:10,346 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646531476257 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646531476257 2022-03-06 02:48:10,364 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646532082240 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646532082240 2022-03-06 02:48:17,978 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 627 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:48:17,978 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 526 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:48:26,642 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 876 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:48:26,807 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 948 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:48:26,807 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 834 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:48:33,441 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1317 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:48:33,606 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1380 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:48:33,606 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 808 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:48:33,606 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 784 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:48:33,606 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 753 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:48:33,606 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 753 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:48:41,093 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1354 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:48:43,327 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 954 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:48:43,359 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 977 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:48:54,090 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1193ms No GCs detected 2022-03-06 02:49:44,649 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 521 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:50:38,953 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1905 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:50:38,953 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 744 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:50:48,126 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 937 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:50:48,126 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 731 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:50:59,727 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 874 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:50:59,727 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 815 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:51:02,105 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1986 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:51:02,105 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1755 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:51:02,105 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1716 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:51:02,171 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1380 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:51:33,881 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1126 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:51:33,881 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1126 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:51:33,881 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1048 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:51:33,881 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1048 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=300.27 KB, freeSize=306.91 MB, max=307.20 MB, blockCount=9, accesses=23797, hits=16853, hitRatio=70.82%, , cachingAccesses=16950, cachingHits=16549, cachingHitsRatio=97.63%, evictions=8361, evicted=0, evictedPerRun=0.0 2022-03-06 02:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 02:51:57,480 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1524ms No GCs detected 2022-03-06 02:52:03,460 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 688 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:52:03,461 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 591 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:52:09,089 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2043 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:52:09,130 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1330 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:52:17,342 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1319 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:52:17,948 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 537 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:52:23,900 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1318ms No GCs detected 2022-03-06 02:52:23,939 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1839 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:52:23,970 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 817 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:52:26,637 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2079 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:52:27,031 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1358 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:52:27,032 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1110 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:52:27,032 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 982 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:52:28,779 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1335 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:52:28,779 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1335 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:52:32,490 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1845ms No GCs detected 2022-03-06 02:52:34,215 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5341 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:52:34,216 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1534 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:52:34,216 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 891 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:52:34,216 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 890 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:52:34,216 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 889 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:52:34,216 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 862 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:52:34,216 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 856 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:52:34,216 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 854 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:52:34,216 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 707 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:52:40,790 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 02:52:44,434 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3207 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:52:44,434 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3113 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:52:44,434 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2660 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:52:50,452 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1421ms No GCs detected 2022-03-06 02:52:50,806 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6763 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:52:50,807 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6372 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:52:50,807 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5550 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:52:50,807 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2277 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:52:50,807 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2274 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:52:51,632 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 550 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:52:54,064 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2543 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:52:54,065 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1702 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:52:54,065 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1231 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:53:01,285 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 581 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:53:01,285 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 544 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:53:04,140 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 711 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:53:11,141 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 02:53:31,161 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2343 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:53:31,161 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2343 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:53:31,161 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2216 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:53:31,161 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1185 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:53:31,182 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 806 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:53:31,182 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 777 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:53:53,511 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1302ms No GCs detected 2022-03-06 02:54:03,026 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 874 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:54:04,188 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646534635195.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646535243512.meta 2022-03-06 02:54:04,189 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646534635195.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646534635195.meta 2022-03-06 02:54:47,913 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 693 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:55:13,190 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3454 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:55:13,729 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2550 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:55:13,729 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2352 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:55:13,729 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1738 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:55:14,899 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1169 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:55:23,608 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2101 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:55:23,608 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1283 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:55:23,608 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 756 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:55:46,697 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2496 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:55:46,697 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1249 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:56:30,879 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 903 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=300.27 KB, freeSize=306.91 MB, max=307.20 MB, blockCount=9, accesses=23819, hits=16875, hitRatio=70.85%, , cachingAccesses=16972, cachingHits=16571, cachingHitsRatio=97.64%, evictions=8391, evicted=0, evictedPerRun=0.0 2022-03-06 02:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 02:57:52,373 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 860 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 02:58:15,438 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646534889273 with entries=1317, filesize=573.56 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646535494292 2022-03-06 02:59:47,656 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 506 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:00:45,787 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 760 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:00:52,874 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 583 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:00:53,578 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 703 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=300.27 KB, freeSize=306.91 MB, max=307.20 MB, blockCount=9, accesses=23841, hits=16897, hitRatio=70.87%, , cachingAccesses=16994, cachingHits=16593, cachingHitsRatio=97.64%, evictions=8420, evicted=0, evictedPerRun=0.0 2022-03-06 03:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 03:02:23,171 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 869 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:02:38,382 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4219 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:02:38,382 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2740 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:02:38,382 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1367 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:03:03,095 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 824 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:04:05,800 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646535243512.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646535845782.meta 2022-03-06 03:04:05,801 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646535243512.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646535243512.meta 2022-03-06 03:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=300.27 KB, freeSize=306.91 MB, max=307.20 MB, blockCount=9, accesses=23863, hits=16919, hitRatio=70.90%, , cachingAccesses=17016, cachingHits=16615, cachingHitsRatio=97.64%, evictions=8450, evicted=0, evictedPerRun=0.0 2022-03-06 03:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 03:08:15,827 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646535494292 with entries=1278, filesize=560.18 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646536095804 2022-03-06 03:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=300.27 KB, freeSize=306.91 MB, max=307.20 MB, blockCount=9, accesses=23885, hits=16941, hitRatio=70.93%, , cachingAccesses=17038, cachingHits=16637, cachingHitsRatio=97.65%, evictions=8480, evicted=0, evictedPerRun=0.0 2022-03-06 03:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 03:12:35,366 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2382ms GC pool 'G1 Young Generation' had collection(s): count=1 time=2546ms 2022-03-06 03:12:35,432 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2613 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:13:42,987 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1242 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:13:43,501 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 659 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:13:44,752 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 615 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:13:46,664 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1413ms No GCs detected 2022-03-06 03:13:47,730 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 577 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:13:47,730 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 524 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:14:06,695 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646535845782.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646536446664.meta 2022-03-06 03:14:06,695 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646535845782.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646535845782.meta 2022-03-06 03:14:18,447 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1000 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:16:11,775 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2037 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:16:11,775 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 897 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=300.27 KB, freeSize=306.91 MB, max=307.20 MB, blockCount=9, accesses=23907, hits=16963, hitRatio=70.95%, , cachingAccesses=17060, cachingHits=16659, cachingHitsRatio=97.65%, evictions=8510, evicted=0, evictedPerRun=0.0 2022-03-06 03:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 03:18:16,718 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646536095804 with entries=1278, filesize=566.94 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646536696698 2022-03-06 03:19:43,082 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb. because 9b121c22e18472616debd0996ee2a4cb/e has an old edit so flush to free WALs after random delay 46690 ms 2022-03-06 03:20:00,554 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 553 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:20:29,772 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 9b121c22e18472616debd0996ee2a4cb 1/1 column families, dataSize=557.96 KB heapSize=1.01 MB 2022-03-06 03:20:29,837 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=557.96 KB at sequenceid=39478 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/.tmp/e/634e8d5502c8434a8928d6a4285b7519 2022-03-06 03:20:29,843 INFO [MemStoreFlusher.0] retry.RetryInvocationHandler: doop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:433) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getBlockLocations over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 03:20:29,850 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/634e8d5502c8434a8928d6a4285b7519, entries=6442, sequenceid=39478, filesize=67.4 K 2022-03-06 03:20:29,851 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~557.96 KB/571348, heapSize ~1.01 MB/1054552, currentSize=0 B/0 for 9b121c22e18472616debd0996ee2a4cb in 79ms, sequenceid=39478, compaction requested=true 2022-03-06 03:20:29,851 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 9b121c22e18472616debd0996ee2a4cb/e in aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb. 2022-03-06 03:20:29,852 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/cc20666269bc451b821afb8deec88cbb, hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/0ef30a00d80845e9be7f4d9e8f84ec7d, hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/e/634e8d5502c8434a8928d6a4285b7519] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/9b121c22e18472616debd0996ee2a4cb/.tmp, totalSize=1.7 M 2022-03-06 03:20:30,084 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 9b121c22e18472616debd0996ee2a4cb#e#compaction#238 average throughput is 60.01 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-06 03:20:30,136 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 9b121c22e18472616debd0996ee2a4cb/e of 9b121c22e18472616debd0996ee2a4cb into 1bea50f5f4ac4ebb9bb84fd36f58b57e(size=1.7 M), total size for store is 1.7 M. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-06 03:20:30,136 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., storeName=9b121c22e18472616debd0996ee2a4cb/e, priority=13, startTime=1646536829851; duration=0sec 2022-03-06 03:21:32,812 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 629 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=300.27 KB, freeSize=306.91 MB, max=307.20 MB, blockCount=9, accesses=24148, hits=16985, hitRatio=70.34%, , cachingAccesses=17082, cachingHits=16681, cachingHitsRatio=97.65%, evictions=8540, evicted=0, evictedPerRun=0.0 2022-03-06 03:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 03:24:06,737 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646536446664.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646537046721.meta 2022-03-06 03:24:06,737 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646536446664.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646536446664.meta 2022-03-06 03:26:03,163 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5. because 7534b3b1812808d2d3c16342fbf347a5/e has an old edit so flush to free WALs after random delay 161568 ms 2022-03-06 03:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=300.27 KB, freeSize=306.91 MB, max=307.20 MB, blockCount=9, accesses=24170, hits=17007, hitRatio=70.36%, , cachingAccesses=17104, cachingHits=16703, cachingHitsRatio=97.66%, evictions=8570, evicted=0, evictedPerRun=0.0 2022-03-06 03:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 03:28:16,769 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646536696698 with entries=1290, filesize=566.38 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646537296749 2022-03-06 03:28:16,769 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646532691388 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646532691388 2022-03-06 03:28:16,773 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646533295103 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646533295103 2022-03-06 03:28:44,731 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 7534b3b1812808d2d3c16342fbf347a5 1/1 column families, dataSize=177.69 KB heapSize=329.58 KB 2022-03-06 03:28:44,819 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=177.69 KB at sequenceid=23138 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/.tmp/e/3a205124d3b5483fbe6f41f389976c1f 2022-03-06 03:28:44,827 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/3a205124d3b5483fbe6f41f389976c1f, entries=2046, sequenceid=23138, filesize=25.0 K 2022-03-06 03:28:44,830 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~177.69 KB/181958, heapSize ~329.56 KB/337472, currentSize=0 B/0 for 7534b3b1812808d2d3c16342fbf347a5 in 98ms, sequenceid=23138, compaction requested=true 2022-03-06 03:28:44,830 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 7534b3b1812808d2d3c16342fbf347a5/e in aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5. 2022-03-06 03:28:44,830 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/04a59ca994914f89ab8fa6dfcc32ec78, hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/88224c4284844dc1800f1cfe6209bcdd, hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/e/3a205124d3b5483fbe6f41f389976c1f] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/7534b3b1812808d2d3c16342fbf347a5/.tmp, totalSize=548.2 K 2022-03-06 03:28:44,914 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 7534b3b1812808d2d3c16342fbf347a5#e#compaction#240 average throughput is 61.53 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-06 03:28:44,994 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 7534b3b1812808d2d3c16342fbf347a5/e of 7534b3b1812808d2d3c16342fbf347a5 into 322f42f3c6654191978377f1a2952905(size=539.7 K), total size for store is 539.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-06 03:28:44,994 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., storeName=7534b3b1812808d2d3c16342fbf347a5/e, priority=13, startTime=1646537324830; duration=0sec 2022-03-06 03:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=300.27 KB, freeSize=306.91 MB, max=307.20 MB, blockCount=9, accesses=24263, hits=17029, hitRatio=70.19%, , cachingAccesses=17126, cachingHits=16725, cachingHitsRatio=97.66%, evictions=8600, evicted=0, evictedPerRun=0.0 2022-03-06 03:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-06 03:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 03:31:48,030 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 894 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:31:48,030 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 648 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:31:52,640 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1045ms No GCs detected 2022-03-06 03:33:56,885 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1136 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:33:56,886 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 935 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:33:58,739 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2745 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:33:58,739 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1247 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:33:58,739 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1247 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:33:58,739 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 950 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:33:58,739 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 950 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:33:58,739 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 710 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:33:58,739 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 609 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:33:59,779 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 955 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:33:59,779 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 955 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:33:59,779 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 947 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:34:09,766 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5578 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:34:11,962 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6377 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:34:11,962 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3587 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:34:12,703 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1449 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:34:12,782 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1162 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:34:12,782 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1162 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:34:12,782 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1011 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:34:13,882 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1908 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:34:13,882 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1908 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:34:13,882 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1001 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:34:14,438 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646537046721.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646537646773.meta 2022-03-06 03:34:14,438 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646537046721.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646537046721.meta 2022-03-06 03:34:18,163 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 782 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:34:18,163 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 707 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:34:22,714 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1287 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:34:25,389 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1009ms No GCs detected 2022-03-06 03:34:25,563 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1682 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:34:27,990 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 727 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:36:06,251 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 685 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:36:07,843 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 841 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:36:09,066 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 853 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:36:09,066 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 671 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:36:16,271 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 6590 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:36:16,280 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3157 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:36:16,280 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1976 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:36:22,977 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 697 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:36:23,781 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 804 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:36:26,601 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1194ms No GCs detected 2022-03-06 03:36:26,633 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 03:36:27,634 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 1 failover attempts. Trying to failover after sleeping for 493ms. 2022-03-06 03:36:33,977 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-06 03:36:36,486 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-06 03:36:36,486 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-06 03:36:36,486 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-06 03:36:36,487 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-06 03:36:36,487 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-06 03:36:36,487 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.sync(AsyncProtobufLogWriter.java:148) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:394) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.consume(AsyncFSWAL.java:581) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-06 03:36:36,450 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 8726ms No GCs detected 2022-03-06 03:36:36,690 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: rg.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 03:36:36,691 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: tNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 2 failover attempts. Trying to failover after sleeping for 1785ms. 2022-03-06 03:36:37,929 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 1 failover attempts. Trying to failover after sleeping for 764ms. 2022-03-06 03:36:38,710 WARN [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: A failover has occurred since the start of call #12482 ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 2022-03-06 03:36:38,710 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: amenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 2 failover attempts. Trying to failover after sleeping for 1429ms. 2022-03-06 03:36:38,711 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: tNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 3 failover attempts. Trying to failover after sleeping for 1822ms. 2022-03-06 03:36:40,259 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646537296749 with entries=1052, filesize=462.17 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646537796528 2022-03-06 03:36:40,259 WARN [Close-WAL-Writer-166] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-06 03:36:40,259 WARN [Close-WAL-Writer-166] wal.AsyncProtobufLogWriter: normal close failed, try recover java.lang.IllegalStateException: should call flush first before calling close at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:510) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:514) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-06 03:36:40,259 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646533436833 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646533436833 2022-03-06 03:36:40,259 INFO [Close-WAL-Writer-166] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646537296749 2022-03-06 03:36:40,270 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646537784177","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":16093,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 03:36:40,271 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646537787523","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":12747,"client":"172.30.41.155:52690","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 03:36:40,271 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646537786600","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":13670,"client":"172.30.41.155:52690","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-06 03:36:40,271 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646537786722","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":13549,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 03:36:40,271 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646537786599","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":13671,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 03:36:40,271 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646537788120","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":12151,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 03:36:40,272 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646537787924","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":12348,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 03:36:40,320 INFO [Close-WAL-Writer-166] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646537296749 after 61ms 2022-03-06 03:36:40,543 WARN [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: A failover has occurred since the start of call #12482 ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 2022-03-06 03:36:44,321 INFO [Close-WAL-Writer-166] util.RecoverLeaseFSUtils: Recovered lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646537296749 after 4062ms 2022-03-06 03:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=300.27 KB, freeSize=306.91 MB, max=307.20 MB, blockCount=9, accesses=24291, hits=17057, hitRatio=70.22%, , cachingAccesses=17154, cachingHits=16753, cachingHitsRatio=97.66%, evictions=8630, evicted=0, evictedPerRun=0.0 2022-03-06 03:36:47,843 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 680 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 03:37:12,977 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb. because ccdacf4b414cb8d1bc548efc77c663cb/e has an old edit so flush to free WALs after random delay 246064 ms 2022-03-06 03:37:21,172 INFO [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] retry.RetryInvocationHandler: doop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getServerDefaults(ClientNamenodeProtocolServerSideTranslatorPB.java:451) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getServerDefaults over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 03:37:34,037 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1858 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:37:34,037 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 908 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:37:34,037 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 907 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:37:34,037 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 907 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:37:34,037 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 906 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:37:35,152 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 993 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:37:35,152 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 720 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:37:35,152 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 564 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:37:43,431 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 568 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:37:45,876 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1685 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:37:58,061 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2238 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:37:58,061 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2070 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:37:58,061 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1757 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:37:58,061 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1095 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:37:58,061 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 796 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:38:01,103 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2291 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:38:01,103 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2221 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:38:01,103 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1284 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:38:01,103 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 872 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:38:01,103 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 723 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:38:01,103 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 697 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:38:01,103 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 582 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:38:16,041 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3114 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:38:16,360 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2272 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:38:16,360 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2140 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:38:16,360 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 866 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:38:18,047 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 922 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:38:18,048 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 635 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:38:28,182 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 979 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:38:28,182 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 877 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:38:28,182 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 877 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:38:30,920 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 539 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:38:30,920 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 517 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:38:36,783 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2601 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:38:36,783 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1182 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:38:40,920 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 996 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:38:42,066 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 830 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:39:36,300 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 677 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:40:25,505 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1041 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:40:36,273 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 667 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:40:56,154 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1378ms No GCs detected 2022-03-06 03:40:57,360 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 5585 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:40:57,360 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4612 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:40:57,360 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1215 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:40:59,688 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3525 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:40:59,688 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3524 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:40:59,688 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3442 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:40:59,782 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1954 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:40:59,783 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1934 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:40:59,784 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 579 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 03:41:15,130 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 03:41:15,585 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-06 03:41:16,198 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-06 03:41:16,200 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-06 03:41:16,200 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-06 03:41:16,200 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-06 03:41:16,205 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: rg.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 03:41:16,206 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 1 failover attempts. Trying to failover after sleeping for 761ms. 2022-03-06 03:41:16,967 WARN [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: A failover has occurred since the start of call #12499 ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 2022-03-06 03:41:17,207 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 1 failover attempts. Trying to failover after sleeping for 845ms. 2022-03-06 03:41:18,053 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: NamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 2 failover attempts. Trying to failover after sleeping for 771ms. 2022-03-06 03:41:18,826 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: amenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 3 failover attempts. Trying to failover after sleeping for 2896ms. 2022-03-06 03:41:19,041 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing ccdacf4b414cb8d1bc548efc77c663cb 1/1 column families, dataSize=302.77 KB heapSize=526.70 KB 2022-03-06 03:41:22,203 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646537796528 with entries=627, filesize=275.83 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646538076200 2022-03-06 03:41:22,203 WARN [Close-WAL-Writer-167] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-06 03:41:22,204 WARN [Close-WAL-Writer-167] wal.AsyncProtobufLogWriter: normal close failed, try recover java.lang.IllegalStateException: should call flush first before calling close at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:510) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:514) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-06 03:41:22,205 INFO [Close-WAL-Writer-167] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646537796528 2022-03-06 03:41:22,206 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646538066952","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":15253,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-06 03:41:22,206 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646538069662","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":12543,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 03:41:22,206 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646538071314","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":10891,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 03:41:22,210 INFO [Close-WAL-Writer-167] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646537796528 after 5ms 2022-03-06 03:41:22,275 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646537646773.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646538082204.meta 2022-03-06 03:41:22,275 WARN [Close-WAL-Writer-170] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-06 03:41:22,275 WARN [Close-WAL-Writer-170] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-06 03:41:22,275 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646537646773.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646537646773.meta 2022-03-06 03:41:22,275 INFO [Close-WAL-Writer-170] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646537646773.meta 2022-03-06 03:41:22,295 INFO [Thread-3991] retry.RetryInvocationHandler: at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:568) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.addBlock over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 03:41:22,296 INFO [Close-WAL-Writer-170] retry.RetryInvocationHandler: .apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.recoverLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 03:41:22,296 INFO [Log-Archiver-0] retry.RetryInvocationHandler: at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setTimes(ClientNamenodeProtocolServerSideTranslatorPB.java:1062) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.setTimes over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 03:41:22,296 WARN [Close-WAL-Writer-170] retry.RetryInvocationHandler: A failover has occurred since the start of call #12512 ClientNamenodeProtocolTranslatorPB.recoverLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 2022-03-06 03:41:22,296 WARN [Log-Archiver-0] retry.RetryInvocationHandler: A failover has occurred since the start of call #12511 ClientNamenodeProtocolTranslatorPB.setTimes over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 2022-03-06 03:41:22,317 INFO [Thread-3991] retry.RetryInvocationHandler: ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:568) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.addBlock over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 1 failover attempts. Trying to failover after sleeping for 818ms. 2022-03-06 03:41:22,317 INFO [Close-WAL-Writer-170] retry.RetryInvocationHandler: menodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.recoverLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 1 failover attempts. Trying to failover after sleeping for 785ms. 2022-03-06 03:41:22,317 INFO [Log-Archiver-0] retry.RetryInvocationHandler: lientNamenodeProtocolServerSideTranslatorPB.setTimes(ClientNamenodeProtocolServerSideTranslatorPB.java:1062) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.setTimes over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 1 failover attempts. Trying to failover after sleeping for 802ms. 2022-03-06 03:41:22,981 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df. because 80a1ffe1dc2d8f258a6367d204dc64df/e has an old edit so flush to free WALs after random delay 68984 ms 2022-03-06 03:41:23,104 INFO [Close-WAL-Writer-170] retry.RetryInvocationHandler: menodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.recoverLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 2 failover attempts. Trying to failover after sleeping for 935ms. 2022-03-06 03:41:23,119 WARN [Log-Archiver-0] retry.RetryInvocationHandler: A failover has occurred since the start of call #12511 ClientNamenodeProtocolTranslatorPB.setTimes over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 2022-03-06 03:41:23,120 INFO [Log-Archiver-0] retry.RetryInvocationHandler: lientNamenodeProtocolServerSideTranslatorPB.setTimes(ClientNamenodeProtocolServerSideTranslatorPB.java:1062) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.setTimes over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 2 failover attempts. Trying to failover after sleeping for 846ms. 2022-03-06 03:41:23,135 WARN [Thread-3991] retry.RetryInvocationHandler: A failover has occurred since the start of call #12510 ClientNamenodeProtocolTranslatorPB.addBlock over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 2022-03-06 03:41:23,136 INFO [Thread-3991] retry.RetryInvocationHandler: ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:568) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.addBlock over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 2 failover attempts. Trying to failover after sleeping for 836ms. 2022-03-06 03:41:23,972 WARN [Thread-3991] retry.RetryInvocationHandler: A failover has occurred since the start of call #12510 ClientNamenodeProtocolTranslatorPB.addBlock over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 2022-03-06 03:41:24,039 WARN [Close-WAL-Writer-170] retry.RetryInvocationHandler: A failover has occurred since the start of call #12512 ClientNamenodeProtocolTranslatorPB.recoverLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 2022-03-06 03:41:24,191 INFO [Close-WAL-Writer-170] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646537646773.meta after 1916ms 2022-03-06 03:41:26,389 INFO [Close-WAL-Writer-167] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646537796528 after 4184ms 2022-03-06 03:41:26,402 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=302.95 KB at sequenceid=15634 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/.tmp/e/7d786dd263fe4f34b3de6587cf651056 2022-03-06 03:41:26,422 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/7d786dd263fe4f34b3de6587cf651056, entries=3043, sequenceid=15634, filesize=31.7 K 2022-03-06 03:41:26,423 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~302.95 KB/310222, heapSize ~527.02 KB/539664, currentSize=402 B/402 for ccdacf4b414cb8d1bc548efc77c663cb in 7382ms, sequenceid=15634, compaction requested=true 2022-03-06 03:41:26,423 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of ccdacf4b414cb8d1bc548efc77c663cb/e in aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb. 2022-03-06 03:41:26,423 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/f0445156166c4938b3ec6eada2467fed, hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/b1e690a811a64b30857392d00d21b91e, hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/e/7d786dd263fe4f34b3de6587cf651056] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/ccdacf4b414cb8d1bc548efc77c663cb/.tmp, totalSize=854.2 K 2022-03-06 03:41:26,575 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: ccdacf4b414cb8d1bc548efc77c663cb#e#compaction#242 average throughput is 61.11 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-06 03:41:26,806 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in ccdacf4b414cb8d1bc548efc77c663cb/e of ccdacf4b414cb8d1bc548efc77c663cb into cfec48cc9f304dc88e0750185a98b166(size=856.9 K), total size for store is 856.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-06 03:41:26,806 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., storeName=ccdacf4b414cb8d1bc548efc77c663cb/e, priority=13, startTime=1646538086423; duration=0sec 2022-03-06 03:41:28,262 WARN [Close-WAL-Writer-170] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646537646773.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646537646773.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-06 03:41:43,719 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1016ms No GCs detected 2022-03-06 03:41:44,441 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=335.80 KB, freeSize=306.87 MB, max=307.20 MB, blockCount=19, accesses=24518, hits=17160, hitRatio=69.99%, , cachingAccesses=17266, cachingHits=16855, cachingHitsRatio=97.62%, evictions=8660, evicted=0, evictedPerRun=0.0 2022-03-06 03:41:46,977 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2517 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:41:46,977 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2513 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 03:41:48,839 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 03:41:48,902 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 4357 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:41:48,902 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3078 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:41:48,902 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1724 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:41:48,902 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1474 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:41:48,903 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 744ms. 2022-03-06 03:41:48,914 INFO [RS_COMPACTED_FILES_DISCHARGER-regionserver/cvp328:16201-7] retry.RetryInvocationHandler: 157) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:714) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.mkdirs over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 03:41:49,647 WARN [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: A failover has occurred since the start of call #12566 ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 2022-03-06 03:41:50,066 INFO [RS_COMPACTED_FILES_DISCHARGER-regionserver/cvp328:16201-7] retry.RetryInvocationHandler: 040) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.rename(ClientNamenodeProtocolServerSideTranslatorPB.java:652) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.rename over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 03:41:50,068 INFO [RS_COMPACTED_FILES_DISCHARGER-regionserver/cvp328:16201-7] retry.RetryInvocationHandler: lPB.ClientNamenodeProtocolServerSideTranslatorPB.rename(ClientNamenodeProtocolServerSideTranslatorPB.java:652) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.rename over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 365ms. 2022-03-06 03:41:50,434 INFO [RS_COMPACTED_FILES_DISCHARGER-regionserver/cvp328:16201-7] retry.RetryInvocationHandler: PB.ClientNamenodeProtocolServerSideTranslatorPB.rename(ClientNamenodeProtocolServerSideTranslatorPB.java:652) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.rename over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 2 failover attempts. Trying to failover after sleeping for 1732ms. 2022-03-06 03:41:57,492 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1114 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:41:57,508 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1128 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:42:06,566 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 976 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:42:24,869 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 660 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:42:31,123 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 660 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:42:31,965 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 80a1ffe1dc2d8f258a6367d204dc64df 1/1 column families, dataSize=287.22 KB heapSize=535.35 KB 2022-03-06 03:42:32,109 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=287.22 KB at sequenceid=28588 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/.tmp/e/2767b9cfe6824e41b38ce71cf991c3e5 2022-03-06 03:42:32,115 INFO [MemStoreFlusher.0] retry.RetryInvocationHandler: doop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:433) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getBlockLocations over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 03:42:32,128 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/2767b9cfe6824e41b38ce71cf991c3e5, entries=3387, sequenceid=28588, filesize=38.1 K 2022-03-06 03:42:32,131 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~287.22 KB/294110, heapSize ~535.34 KB/548184, currentSize=0 B/0 for 80a1ffe1dc2d8f258a6367d204dc64df in 166ms, sequenceid=28588, compaction requested=true 2022-03-06 03:42:32,131 INFO [regionserver/cvp328:16201-longCompactions-0] regionserver.HRegion: Starting compaction of 80a1ffe1dc2d8f258a6367d204dc64df/e in aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df. 2022-03-06 03:42:32,131 INFO [regionserver/cvp328:16201-longCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/e9c833cc814f4217b82557bcd6b76ed4, hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/9a9d9d397a6e47968a8792a1d9f97fb3, hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/e/2767b9cfe6824e41b38ce71cf991c3e5] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/80a1ffe1dc2d8f258a6367d204dc64df/.tmp, totalSize=950.3 K 2022-03-06 03:42:32,286 INFO [regionserver/cvp328:16201-longCompactions-0] throttle.PressureAwareThroughputController: 80a1ffe1dc2d8f258a6367d204dc64df#e#compaction#244 average throughput is 60.62 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-06 03:42:33,812 INFO [regionserver/cvp328:16201-longCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 80a1ffe1dc2d8f258a6367d204dc64df/e of 80a1ffe1dc2d8f258a6367d204dc64df into 0a37cb55bebf454f83c16a13ed163368(size=950.0 K), total size for store is 950.0 K. This selection was in queue for 0sec, and took 1sec to execute. 2022-03-06 03:42:33,812 INFO [regionserver/cvp328:16201-longCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., storeName=80a1ffe1dc2d8f258a6367d204dc64df/e, priority=13, startTime=1646538152131; duration=1sec 2022-03-06 03:42:37,723 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 744 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:42:39,229 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1255ms No GCs detected 2022-03-06 03:42:40,637 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 868 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:42:42,055 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 758 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:42:45,500 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3754 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:42:44,723 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2586ms No GCs detected 2022-03-06 03:42:49,726 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 03:42:57,414 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-06 03:43:00,379 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-06 03:43:00,379 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-06 03:43:00,379 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-06 03:43:00,379 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-06 03:43:00,379 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-06 03:43:00,379 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-06 03:43:00,154 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1715ms No GCs detected 2022-03-06 03:43:10,474 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:568) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.addBlock over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 03:43:10,476 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:568) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.addBlock over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 726ms. 2022-03-06 03:43:11,204 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: lientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:568) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.addBlock over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 2 failover attempts. Trying to failover after sleeping for 1298ms. 2022-03-06 03:43:12,994 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391. because 0a764e51258807790269e46c55b99391/e has an old edit so flush to free WALs after random delay 253958 ms 2022-03-06 03:43:12,994 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. because 7d1b53259f1d8280a47f7704f98f2c9f/e has an old edit so flush to free WALs after random delay 141947 ms 2022-03-06 03:43:20,162 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1318ms No GCs detected 2022-03-06 03:43:22,164 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1501ms No GCs detected 2022-03-06 03:43:25,442 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646538076200 with entries=235, filesize=105.60 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646538180379 2022-03-06 03:43:25,443 WARN [Close-WAL-Writer-168] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-06 03:43:25,443 WARN [Close-WAL-Writer-168] wal.AsyncProtobufLogWriter: normal close failed, try recover java.lang.IllegalStateException: should call flush first before calling close at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkState(Preconditions.java:510) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:514) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-06 03:43:25,443 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646533677670 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646533677670 2022-03-06 03:43:25,443 INFO [Close-WAL-Writer-168] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646538076200 2022-03-06 03:43:25,727 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646538175850","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":29877,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 03:43:25,727 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646538172307","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":33420,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 03:43:25,727 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646538176765","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":28962,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":3,"class":"HRegionServer","multi.mutations":1} 2022-03-06 03:43:25,727 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646538175746","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":29981,"client":"172.30.41.155:52690","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 03:43:25,727 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646538177939","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":27788,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 03:43:25,727 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646538177869","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":27858,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 03:43:25,727 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646538171412","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":34315,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 03:43:25,727 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646538175132","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":30595,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 03:43:25,727 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646538174208","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":31519,"client":"172.30.41.155:52690","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 03:43:25,727 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646538175747","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":29980,"client":"172.30.41.155:52690","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 03:43:25,748 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646538178111","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":27635,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 03:43:25,748 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646538180382","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":25365,"client":"172.30.41.155:52690","multi.service_calls":0,"queuetimems":135,"class":"HRegionServer","multi.mutations":1} 2022-03-06 03:43:25,748 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646538177939","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":27808,"client":"172.31.0.219:42696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 03:43:25,748 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646538178016","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":27731,"client":"172.30.41.155:52690","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 03:43:25,748 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646538178017","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":27730,"client":"172.30.41.155:52690","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 03:43:25,748 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646538178017","responsesize":"10","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":27730,"client":"172.30.41.155:52690","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 03:43:28,648 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2476 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:43:28,681 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2510 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:43:28,681 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2510 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:43:28,681 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2510 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:43:28,681 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2510 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:43:28,681 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2510 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:43:28,681 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2510 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:43:28,753 INFO [Close-WAL-Writer-168] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646538076200 after 3310ms 2022-03-06 03:43:30,808 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 562 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:43:30,857 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:568) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.addBlock over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 03:43:30,863 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:568) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.addBlock over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 1 failover attempts. Trying to failover after sleeping for 596ms. 2022-03-06 03:43:31,461 INFO [regionserver/cvp328:16201.logRoller] retry.RetryInvocationHandler: ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:568) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.addBlock over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 2 failover attempts. Trying to failover after sleeping for 824ms. 2022-03-06 03:43:32,753 INFO [Close-WAL-Writer-168] util.RecoverLeaseFSUtils: Recovered lease, attempt=1 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646538076200 after 7310ms 2022-03-06 03:43:33,589 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646538082204.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646538205443.meta 2022-03-06 03:43:33,589 WARN [Close-WAL-Writer-171] wal.AbstractProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush0(FanOutOneBlockAsyncDFSOutput.java:420) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.flush(FanOutOneBlockAsyncDFSOutput.java:509) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.lambda$writeWALTrailerAndMagic$3(AsyncProtobufLogWriter.java:231) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:187) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeWALTrailerAndMagic(AsyncProtobufLogWriter.java:222) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.writeWALTrailer(AbstractProtobufLogWriter.java:235) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:157) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-06 03:43:33,589 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646538082204.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646538082204.meta 2022-03-06 03:43:33,589 WARN [Close-WAL-Writer-171] wal.AsyncProtobufLogWriter: normal close failed, try recover java.io.IOException: stream already broken at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:517) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.close(FanOutOneBlockAsyncDFSOutput.java:565) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:158) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-03-06 03:43:33,589 INFO [Close-WAL-Writer-171] util.RecoverLeaseFSUtils: Recover lease on dfs file /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646538082204.meta 2022-03-06 03:43:37,686 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3498 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:43:37,686 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2945 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:43:37,686 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2040 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:43:37,690 INFO [Close-WAL-Writer-171] util.RecoverLeaseFSUtils: Failed to recover lease, attempt=0 on file=/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646538082204.meta after 4101ms 2022-03-06 03:43:42,512 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3462 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:43:42,512 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3457 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:43:42,512 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2413 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:43:43,416 WARN [Close-WAL-Writer-171] wal.AsyncFSWAL: close old writer failed java.io.FileNotFoundException: File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646538082204.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:888) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:308) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverLease(RecoverLeaseFSUtils.java:172) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverDFSFileLease(RecoverLeaseFSUtils.java:107) at org.apache.hadoop.hbase.util.RecoverLeaseFSUtils.recoverFileLease(RecoverLeaseFSUtils.java:62) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.recoverAndClose(FanOutOneBlockAsyncDFSOutput.java:555) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.close(AsyncProtobufLogWriter.java:161) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.lambda$closeWriter$5(AsyncFSWAL.java:698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646538082204.meta at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2543) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:848) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:755) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1562) at org.apache.hadoop.ipc.Client.call(Client.java:1508) at org.apache.hadoop.ipc.Client.call(Client.java:1405) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) at com.sun.proxy.$Proxy18.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:700) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy19.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:376) at com.sun.proxy.$Proxy20.recoverLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:886) ... 13 more 2022-03-06 03:43:45,891 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1220 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:43:45,891 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1218 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:43:51,965 INFO [RS_COMPACTED_FILES_DISCHARGER-regionserver/cvp328:16201-2] retry.RetryInvocationHandler: rg.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 03:43:51,968 INFO [RS_COMPACTED_FILES_DISCHARGER-regionserver/cvp328:16201-2] retry.RetryInvocationHandler: NamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 306ms. 2022-03-06 03:43:52,284 INFO [RS_COMPACTED_FILES_DISCHARGER-regionserver/cvp328:16201-2] retry.RetryInvocationHandler: amenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:973) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 2 failover attempts. Trying to failover after sleeping for 1250ms. 2022-03-06 03:44:08,453 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1111ms No GCs detected 2022-03-06 03:44:10,535 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1968 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:44:12,320 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 580 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:44:34,785 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 599 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:44:42,991 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e. because 4333b489291b1f77671b8f2e78779a1e/__status_family___ has an old edit so flush to free WALs after random delay 258753 ms 2022-03-06 03:45:00,037 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1281 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:45:00,037 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1279 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:45:11,770 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 582 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:45:11,807 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 581 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:45:34,941 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 7d1b53259f1d8280a47f7704f98f2c9f 1/1 column families, dataSize=662.49 KB heapSize=1.11 MB 2022-03-06 03:45:35,536 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=662.49 KB at sequenceid=26754 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/.tmp/e/5f916dcea326446b9452a6e6eda2f8c7 2022-03-06 03:45:35,567 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/5f916dcea326446b9452a6e6eda2f8c7, entries=6389, sequenceid=26754, filesize=63.8 K 2022-03-06 03:45:35,586 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~662.49 KB/678391, heapSize ~1.11 MB/1163640, currentSize=0 B/0 for 7d1b53259f1d8280a47f7704f98f2c9f in 645ms, sequenceid=26754, compaction requested=true 2022-03-06 03:45:35,586 INFO [regionserver/cvp328:16201-longCompactions-0] regionserver.HRegion: Starting compaction of 7d1b53259f1d8280a47f7704f98f2c9f/e in aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. 2022-03-06 03:45:35,586 INFO [regionserver/cvp328:16201-longCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/ef63d94635f14827ad8baf32533d856b, hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/06b95ff5019b4759bfcd2ae50d6bf3e5, hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/e/5f916dcea326446b9452a6e6eda2f8c7] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/7d1b53259f1d8280a47f7704f98f2c9f/.tmp, totalSize=1.8 M 2022-03-06 03:45:36,177 INFO [regionserver/cvp328:16201-longCompactions-0] throttle.PressureAwareThroughputController: 7d1b53259f1d8280a47f7704f98f2c9f#e#compaction#246 average throughput is 29.96 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-06 03:45:36,384 INFO [regionserver/cvp328:16201-longCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 7d1b53259f1d8280a47f7704f98f2c9f/e of 7d1b53259f1d8280a47f7704f98f2c9f into ad954a2afe7e4e8686ca8c13fdbacd7f(size=1.9 M), total size for store is 1.9 M. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-06 03:45:36,384 INFO [regionserver/cvp328:16201-longCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., storeName=7d1b53259f1d8280a47f7704f98f2c9f/e, priority=13, startTime=1646538335586; duration=0sec 2022-03-06 03:46:16,776 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1809 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:46:16,776 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1062 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=400.16 KB, freeSize=306.81 MB, max=307.20 MB, blockCount=17, accesses=24939, hits=17205, hitRatio=68.99%, , cachingAccesses=17316, cachingHits=16898, cachingHitsRatio=97.59%, evictions=8690, evicted=0, evictedPerRun=0.0 2022-03-06 03:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 03:47:26,952 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 0a764e51258807790269e46c55b99391 1/1 column families, dataSize=955.45 KB heapSize=1.67 MB 2022-03-06 03:47:27,008 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed memstore data size=955.45 KB at sequenceid=55684 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/.tmp/e/1ae1dcc2c6bf4a00a02b37bb188af303 2022-03-06 03:47:27,016 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/1ae1dcc2c6bf4a00a02b37bb188af303, entries=10263, sequenceid=55684, filesize=93.0 K 2022-03-06 03:47:27,017 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished flush of dataSize ~955.45 KB/978384, heapSize ~1.67 MB/1756256, currentSize=0 B/0 for 0a764e51258807790269e46c55b99391 in 65ms, sequenceid=55684, compaction requested=true 2022-03-06 03:47:27,017 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 0a764e51258807790269e46c55b99391/e in aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391. 2022-03-06 03:47:27,017 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/7dbeb943396c45ca8019755a770a44e2, hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/c86941e1554349f6b32f37ca3c8dd508, hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/e/1ae1dcc2c6bf4a00a02b37bb188af303] into tmpdir=hdfs://mycluster/hbase/data/default/aeris_v2/0a764e51258807790269e46c55b99391/.tmp, totalSize=2.6 M 2022-03-06 03:47:27,418 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 0a764e51258807790269e46c55b99391#e#compaction#248 average throughput is 56.50 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-06 03:47:27,477 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 0a764e51258807790269e46c55b99391/e of 0a764e51258807790269e46c55b99391 into bc0923b89dac4cb296061ba6220d23e9(size=2.7 M), total size for store is 2.7 M. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-06 03:47:27,477 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391., storeName=0a764e51258807790269e46c55b99391/e, priority=13, startTime=1646538447017; duration=0sec 2022-03-06 03:49:01,744 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 4333b489291b1f77671b8f2e78779a1e 1/1 column families, dataSize=8.12 KB heapSize=12.95 KB 2022-03-06 03:49:01,786 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed memstore data size=8.12 KB at sequenceid=1591 (bloomFilter=true), to=hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/.tmp/__status_family___/40f2982fd83945cdaf1ca6c3723f252d 2022-03-06 03:49:01,795 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/40f2982fd83945cdaf1ca6c3723f252d, entries=3, sequenceid=1591, filesize=5.1 K 2022-03-06 03:49:01,795 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished flush of dataSize ~8.12 KB/8314, heapSize ~12.93 KB/13240, currentSize=0 B/0 for 4333b489291b1f77671b8f2e78779a1e in 51ms, sequenceid=1591, compaction requested=true 2022-03-06 03:49:01,796 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HRegion: Starting compaction of 4333b489291b1f77671b8f2e78779a1e/__status_family___ in __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e. 2022-03-06 03:49:01,796 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Starting compaction of [hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/7115309d12c142b99c9ca34c5d411082, hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/dbc8d0a0e30941f08abf4eb28c7a90da, hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/__status_family___/40f2982fd83945cdaf1ca6c3723f252d] into tmpdir=hdfs://mycluster/hbase/data/default/__test-table_cvp328.sjc.aristanetworks.com__/4333b489291b1f77671b8f2e78779a1e/.tmp, totalSize=15.4 K 2022-03-06 03:49:01,815 INFO [regionserver/cvp328:16201-shortCompactions-0] throttle.PressureAwareThroughputController: 4333b489291b1f77671b8f2e78779a1e#__status_family___#compaction#250 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2022-03-06 03:49:01,858 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.HStore: Completed compaction of 3 (all) file(s) in 4333b489291b1f77671b8f2e78779a1e/__status_family___ of 4333b489291b1f77671b8f2e78779a1e into d93628d7083d4db5bafb0c970d7a55d2(size=5.2 K), total size for store is 5.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2022-03-06 03:49:01,858 INFO [regionserver/cvp328:16201-shortCompactions-0] regionserver.CompactSplit: Completed compaction region=__test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., storeName=4333b489291b1f77671b8f2e78779a1e/__status_family___, priority=13, startTime=1646538541796; duration=0sec 2022-03-06 03:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=494.22 KB, freeSize=306.72 MB, max=307.20 MB, blockCount=20, accesses=25346, hits=17250, hitRatio=68.06%, , cachingAccesses=17366, cachingHits=16941, cachingHitsRatio=97.55%, evictions=8720, evicted=0, evictedPerRun=0.0 2022-03-06 03:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 03:52:10,256 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2012 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:53:03,621 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646538180379 with entries=1370, filesize=587.17 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646538783596 2022-03-06 03:53:03,621 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646534282210 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646534282210 2022-03-06 03:53:03,631 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646534889273 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646534889273 2022-03-06 03:53:03,640 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646535494292 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646535494292 2022-03-06 03:53:03,648 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646536095804 to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646536095804 2022-03-06 03:53:33,799 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646538205443.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646538813624.meta 2022-03-06 03:53:33,800 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646538205443.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646538205443.meta 2022-03-06 03:55:25,064 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1575ms No GCs detected 2022-03-06 03:55:26,943 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3849 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:55:26,943 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2751 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:55:26,945 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1067 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:55:26,945 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 811 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:55:54,280 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2877 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:55:54,336 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2101 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:56:01,696 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1124ms No GCs detected 2022-03-06 03:56:14,335 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3152 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:56:14,352 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3043 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:56:14,352 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1442 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:56:14,352 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1052 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=538.89 KB, freeSize=306.67 MB, max=307.20 MB, blockCount=22, accesses=25394, hits=17296, hitRatio=68.11%, , cachingAccesses=17414, cachingHits=16987, cachingHitsRatio=97.55%, evictions=8750, evicted=0, evictedPerRun=0.0 2022-03-06 03:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 03:57:53,209 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 956 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:58:09,070 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 3348 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:58:09,131 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1352 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:58:09,131 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 538 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:58:09,131 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 529 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 03:58:21,849 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1430ms No GCs detected 2022-03-06 03:58:41,142 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1394 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 04:00:15,588 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 573 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 04:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=25432, hits=17333, hitRatio=68.15%, , cachingAccesses=17452, cachingHits=17024, cachingHitsRatio=97.55%, evictions=8780, evicted=0, evictedPerRun=0.0 2022-03-06 04:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 04:01:56,974 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2791 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 04:01:56,974 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 2477 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 04:01:56,974 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1193 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 04:01:56,974 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1179 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 04:01:56,974 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 978 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK]] 2022-03-06 04:03:03,885 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646538783596 with entries=1285, filesize=562.39 KB; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.1646539383858 2022-03-06 04:03:34,165 INFO [regionserver/cvp328:16201.logRoller] wal.AbstractFSWAL: Rolled WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646538813624.meta with entries=0, filesize=83 B; new WAL /hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646539413886.meta 2022-03-06 04:03:34,165 INFO [Log-Archiver-0] wal.AbstractFSWAL: Archiving hdfs://mycluster/hbase/WALs/cvp328.sjc.aristanetworks.com,16201,1646451103340/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646538813624.meta to hdfs://mycluster/hbase/oldWALs/cvp328.sjc.aristanetworks.com%2C16201%2C1646451103340.meta.1646538813624.meta 2022-03-06 04:04:01,479 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 716 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 04:04:01,479 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 524 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 04:04:03,577 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 787 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 04:04:04,169 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1379 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 04:04:04,169 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 759 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 04:04:04,164 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1714ms No GCs detected 2022-03-06 04:05:58,587 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 519 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 04:06:43,076 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1362ms No GCs detected 2022-03-06 04:06:43,135 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1760 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 04:06:43,135 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 1755 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 04:06:43,135 INFO [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AbstractFSWAL: Slow sync cost: 880 ms, current pipeline: [DatanodeInfoWithStorage[172.30.41.118:15010,DS-3dff1f72-169a-438f-a8ae-d1566f7562f1,DISK], DatanodeInfoWithStorage[172.30.41.155:15010,DS-8070a27d-1f1d-4249-b691-f82a6c68a2b1,DISK], DatanodeInfoWithStorage[172.31.0.219:15010,DS-41b9f8d0-139e-49ba-8f71-64469805731e,DISK]] 2022-03-06 04:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=25454, hits=17355, hitRatio=68.18%, , cachingAccesses=17474, cachingHits=17046, cachingHitsRatio=97.55%, evictions=8810, evicted=0, evictedPerRun=0.0 2022-03-06 04:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 04:06:57,396 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-06 04:06:57,397 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-06 04:06:57,397 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-06 04:06:57,397 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-06 04:06:57,397 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-06 04:06:57,397 WARN [AsyncFSWAL-0-hdfs://mycluster/hbase] wal.AsyncFSWAL: sync failed java.io.IOException: Timeout(10000ms) waiting for response at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.lambda$userEventTriggered$4(FanOutOneBlockAsyncDFSOutput.java:300) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.failed(FanOutOneBlockAsyncDFSOutput.java:233) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.access$300(FanOutOneBlockAsyncDFSOutput.java:98) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput$AckHandler.userEventTriggered(FanOutOneBlockAsyncDFSOutput.java:299) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelIdle(IdleStateHandler.java:371) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:504) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:476) at org.apache.hbase.thirdparty.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at org.apache.hbase.thirdparty.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2022-03-06 04:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=26190, hits=18091, hitRatio=69.08%, , cachingAccesses=18210, cachingHits=17782, cachingHitsRatio=97.65%, evictions=8840, evicted=0, evictedPerRun=0.0 2022-03-06 04:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 04:11:52,820 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646539612815","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300004,"client":"172.30.41.155:52690","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:11:54,534 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646539614533","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:52690","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:11:55,710 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646539615710","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:52690","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:11:56,030 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646539616029","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:11:57,000 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646539616999","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:52690","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:11:57,200 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646539617199","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:52690","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:11:57,200 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646539617199","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:52690","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:11:57,923 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646539617923","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:11:58,115 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646539618115","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:11:59,727 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646539619726","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:52690","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:11:59,899 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646539619898","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.155:52234","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 04:12:00,389 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646539620388","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:12:00,504 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646539620503","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:52690","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:12:00,721 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646539620720","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:12:00,930 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646539620930","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:52690","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:12:02,123 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646539622122","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:54152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:12:16,579 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3996ms No GCs detected 2022-03-06 04:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=27046, hits=18947, hitRatio=70.05%, , cachingAccesses=19066, cachingHits=18638, cachingHitsRatio=97.76%, evictions=8869, evicted=0, evictedPerRun=0.0 2022-03-06 04:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 04:16:52,822 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646539912821","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:55366","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:16:54,535 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646539914534","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:37032","multi.service_calls":0,"queuetimems":1713,"class":"HRegionServer","multi.mutations":2} 2022-03-06 04:16:55,711 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646539915710","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:45898","multi.service_calls":0,"queuetimems":2889,"class":"HRegionServer","multi.mutations":3} 2022-03-06 04:16:57,000 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646539917000","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:37032","multi.service_calls":0,"queuetimems":4179,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:16:57,201 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646539917200","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:37032","multi.service_calls":0,"queuetimems":1169,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:16:57,954 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646539917953","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300001,"client":"172.30.41.118:37030","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:17:00,551 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646539920389","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300162,"client":"172.30.41.155:55366","multi.service_calls":0,"queuetimems":4358,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:17:01,067 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646539920930","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300137,"client":"172.30.41.155:55364","multi.service_calls":0,"queuetimems":4869,"class":"HRegionServer","multi.mutations":2} 2022-03-06 04:17:22,841 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646539942840","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:45900","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:17:22,841 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646539942840","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:45900","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:17:22,841 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646539942840","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:45900","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:20:33,003 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb. because 9b121c22e18472616debd0996ee2a4cb/e has an old edit so flush to free WALs after random delay 215376 ms 2022-03-06 04:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=27783, hits=19684, hitRatio=70.85%, , cachingAccesses=19803, cachingHits=19375, cachingHitsRatio=97.84%, evictions=8899, evicted=0, evictedPerRun=0.0 2022-03-06 04:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 04:21:52,824 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646540212823","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:55406","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:21:54,536 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646540214535","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.30.41.155:55408","multi.service_calls":0,"queuetimems":1680,"class":"HRegionServer","multi.mutations":2} 2022-03-06 04:21:55,712 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646540215711","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:55406","multi.service_calls":0,"queuetimems":1160,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:21:56,033 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646539916031","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.155:55366","multi.service_calls":0,"queuetimems":3210,"class":"HRegionServer","multi.mutations":3} 2022-03-06 04:21:57,201 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646539917200","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:45898","multi.service_calls":0,"queuetimems":4374,"class":"HRegionServer","multi.mutations":2} 2022-03-06 04:21:57,201 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646540217201","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:55406","multi.service_calls":0,"queuetimems":1473,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:22:00,552 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646540220551","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:45920","multi.service_calls":0,"queuetimems":7727,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:22:22,842 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646540242841","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:45920","multi.service_calls":0,"queuetimems":30017,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:24:08,380 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 9b121c22e18472616debd0996ee2a4cb 1/1 column families, dataSize=427.96 KB heapSize=791.84 KB 2022-03-06 04:24:12,988 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb. because 9b121c22e18472616debd0996ee2a4cb/e has an old edit so flush to free WALs after random delay 164330 ms 2022-03-06 04:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=28555, hits=20456, hitRatio=71.64%, , cachingAccesses=20575, cachingHits=20147, cachingHitsRatio=97.92%, evictions=8929, evicted=0, evictedPerRun=0.0 2022-03-06 04:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 04:26:52,827 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646540512825","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.155:55428","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:26:54,536 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646540514536","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:55428","multi.service_calls":0,"queuetimems":1711,"class":"HRegionServer","multi.mutations":2} 2022-03-06 04:26:57,002 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646540217001","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:55406","multi.service_calls":0,"queuetimems":4178,"class":"HRegionServer","multi.mutations":4} 2022-03-06 04:26:57,202 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646540517201","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:45962","multi.service_calls":0,"queuetimems":1166,"class":"HRegionServer","multi.mutations":2} 2022-03-06 04:26:57,955 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646540217954","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:37084","multi.service_calls":0,"queuetimems":952,"class":"HRegionServer","multi.mutations":2} 2022-03-06 04:27:00,069 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646539919899","responsesize":"19835","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 12 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900169,"client":"172.31.0.219:45898","multi.service_calls":0,"queuetimems":7073,"class":"HRegionServer","multi.mutations":12} 2022-03-06 04:27:00,506 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646539920504","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 7 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.155:55366","multi.service_calls":0,"queuetimems":4473,"class":"HRegionServer","multi.mutations":7} 2022-03-06 04:27:02,987 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb. because 9b121c22e18472616debd0996ee2a4cb/e has an old edit so flush to free WALs after random delay 194534 ms 2022-03-06 04:27:22,842 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646540242841","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:55406","multi.service_calls":0,"queuetimems":25839,"class":"HRegionServer","multi.mutations":2} 2022-03-06 04:27:22,842 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646540242841","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 5 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:55406","multi.service_calls":0,"queuetimems":25839,"class":"HRegionServer","multi.mutations":5} 2022-03-06 04:27:22,950 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646540542949","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:37148","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:28:53,000 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5. because 7534b3b1812808d2d3c16342fbf347a5/e has an old edit so flush to free WALs after random delay 276417 ms 2022-03-06 04:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=29273, hits=21174, hitRatio=72.33%, , cachingAccesses=21293, cachingHits=20865, cachingHitsRatio=97.99%, evictions=8959, evicted=0, evictedPerRun=0.0 2022-03-06 04:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-06 04:31:48,027 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 04:31:54,537 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646540814537","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:55446","multi.service_calls":0,"queuetimems":1709,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:31:57,003 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646540817003","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:55446","multi.service_calls":0,"queuetimems":4175,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:31:57,203 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646540817202","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:55446","multi.service_calls":0,"queuetimems":4374,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:31:57,956 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646540817955","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:55446","multi.service_calls":0,"queuetimems":5127,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:32:00,506 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646540820506","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:55446","multi.service_calls":0,"queuetimems":7678,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:32:00,844 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646539920721","responsesize":"23142","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 14 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200123,"client":"172.30.41.118:37032","multi.service_calls":0,"queuetimems":4690,"class":"HRegionServer","multi.mutations":14} 2022-03-06 04:32:01,069 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646540221067","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.118:37084","multi.service_calls":0,"queuetimems":8243,"class":"HRegionServer","multi.mutations":6} 2022-03-06 04:32:22,843 ERROR [MemStoreFlusher.1] regionserver.MemStoreFlusher: Cache flush failed for region aeris_v2,\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb. org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync result after 300000 ms for txid=389018, WAL system stuck? at org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:174) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:800) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:640) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:617) at org.apache.hadoop.hbase.regionserver.HRegion.doSyncOfUnflushedWALChanges(HRegion.java:2927) at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2864) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2722) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2696) at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2569) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:617) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:586) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$1000(MemStoreFlusher.java:68) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:360) 2022-03-06 04:32:22,843 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646540542842","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:45962","multi.service_calls":0,"queuetimems":30016,"class":"HRegionServer","multi.mutations":3} 2022-03-06 04:32:22,843 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646540516034","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":626809,"client":"172.30.41.118:37142","multi.service_calls":0,"queuetimems":3209,"class":"HRegionServer","multi.mutations":2} 2022-03-06 04:32:22,843 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646540812828","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":330015,"client":"172.30.41.118:37192","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:32:22,843 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646540842842","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:55446","multi.service_calls":0,"queuetimems":30014,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:32:22,843 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646540520552","responsesize":"11572","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":622291,"client":"172.31.0.219:45962","multi.service_calls":0,"queuetimems":7726,"class":"HRegionServer","multi.mutations":7} 2022-03-06 04:32:22,844 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646540842842","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.155:55446","multi.service_calls":0,"queuetimems":30014,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:32:22,951 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646540842950","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:55446","multi.service_calls":0,"queuetimems":30122,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:33:29,418 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 7534b3b1812808d2d3c16342fbf347a5 1/1 column families, dataSize=107.72 KB heapSize=199.95 KB 2022-03-06 04:33:33,318 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5. because 7534b3b1812808d2d3c16342fbf347a5/e has an old edit so flush to free WALs after random delay 161020 ms 2022-03-06 04:36:23,191 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5. because 7534b3b1812808d2d3c16342fbf347a5/e has an old edit so flush to free WALs after random delay 78477 ms 2022-03-06 04:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=30022, hits=21923, hitRatio=73.02%, , cachingAccesses=22042, cachingHits=21614, cachingHitsRatio=98.06%, evictions=8989, evicted=0, evictedPerRun=0.0 2022-03-06 04:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 04:37:00,070 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646540820069","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 9 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:37192","multi.service_calls":0,"queuetimems":7241,"class":"HRegionServer","multi.mutations":9} 2022-03-06 04:37:22,844 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646541142843","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.155:55486","multi.service_calls":0,"queuetimems":28274,"class":"HRegionServer","multi.mutations":2} 2022-03-06 04:37:22,844 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646541142843","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:37244","multi.service_calls":0,"queuetimems":25839,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:37:22,844 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646541142843","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:37244","multi.service_calls":0,"queuetimems":25838,"class":"HRegionServer","multi.mutations":2} 2022-03-06 04:37:22,844 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646540515712","responsesize":"18183","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 11 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":927132,"client":"172.30.41.118:37142","multi.service_calls":0,"queuetimems":2886,"class":"HRegionServer","multi.mutations":11} 2022-03-06 04:37:22,845 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646541142844","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:46018","multi.service_calls":0,"queuetimems":25839,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:37:22,874 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646541142874","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300000,"client":"172.30.41.118:37246","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:37:22,951 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646541142951","responsesize":"11569","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:46018","multi.service_calls":0,"queuetimems":25946,"class":"HRegionServer","multi.mutations":7} 2022-03-06 04:37:27,012 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646541147012","responsesize":"6613","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:55490","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-06 04:39:51,770 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1385ms No GCs detected 2022-03-06 04:39:54,309 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1145ms No GCs detected 2022-03-06 04:41:22,990 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb. because ccdacf4b414cb8d1bc548efc77c663cb/e has an old edit so flush to free WALs after random delay 228502 ms 2022-03-06 04:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=30830, hits=22731, hitRatio=73.73%, , cachingAccesses=22850, cachingHits=22422, cachingHitsRatio=98.13%, evictions=9019, evicted=0, evictedPerRun=0.0 2022-03-06 04:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 04:41:54,540 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646541114538","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.155:55484","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":6} 2022-03-06 04:42:00,082 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646541420071","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300011,"client":"172.30.41.118:54986","multi.service_calls":0,"queuetimems":46,"class":"HRegionServer","multi.mutations":2} 2022-03-06 04:42:00,845 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646541120844","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:55484","multi.service_calls":0,"queuetimems":3840,"class":"HRegionServer","multi.mutations":4} 2022-03-06 04:42:01,070 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646541121069","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:55484","multi.service_calls":0,"queuetimems":4064,"class":"HRegionServer","multi.mutations":2} 2022-03-06 04:42:22,844 ERROR [MemStoreFlusher.1] regionserver.MemStoreFlusher: Cache flush failed for region aeris_v2,\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5. org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync result after 300000 ms for txid=389074, WAL system stuck? at org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:174) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:800) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:640) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:617) at org.apache.hadoop.hbase.regionserver.HRegion.doSyncOfUnflushedWALChanges(HRegion.java:2927) at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2864) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2722) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2696) at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2569) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:617) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:586) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$1000(MemStoreFlusher.java:68) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:360) 2022-03-06 04:42:22,844 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646541442844","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:37292","multi.service_calls":0,"queuetimems":22773,"class":"HRegionServer","multi.mutations":2} 2022-03-06 04:42:22,952 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646541442951","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:37292","multi.service_calls":0,"queuetimems":22418,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:42:23,061 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646541442844","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300217,"client":"172.31.0.219:46040","multi.service_calls":0,"queuetimems":22772,"class":"HRegionServer","multi.mutations":2} 2022-03-06 04:42:24,124 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646541117203","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":626921,"client":"172.31.0.219:46018","multi.service_calls":0,"queuetimems":2665,"class":"HRegionServer","multi.mutations":2} 2022-03-06 04:42:24,125 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1012ms No GCs detected 2022-03-06 04:42:24,130 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646541442845","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":301285,"client":"172.30.41.155:55524","multi.service_calls":0,"queuetimems":22349,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:42:27,049 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646541447013","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300036,"client":"172.30.41.155:55524","multi.service_calls":0,"queuetimems":25536,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:42:32,976 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df. because 80a1ffe1dc2d8f258a6367d204dc64df/e has an old edit so flush to free WALs after random delay 101010 ms 2022-03-06 04:44:13,986 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 80a1ffe1dc2d8f258a6367d204dc64df 1/1 column families, dataSize=112.80 KB heapSize=210.33 KB 2022-03-06 04:44:23,492 INFO [regionserver/cvp328:16201.Chore.3] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df. because 80a1ffe1dc2d8f258a6367d204dc64df/e has an old edit so flush to free WALs after random delay 196730 ms 2022-03-06 04:45:11,493 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing ccdacf4b414cb8d1bc548efc77c663cb 1/1 column families, dataSize=119.72 KB heapSize=208.03 KB 2022-03-06 04:45:12,976 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb. because ccdacf4b414cb8d1bc548efc77c663cb/e has an old edit so flush to free WALs after random delay 41943 ms 2022-03-06 04:45:43,627 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. because 7d1b53259f1d8280a47f7704f98f2c9f/e has an old edit so flush to free WALs after random delay 4228 ms 2022-03-06 04:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=31710, hits=23611, hitRatio=74.46%, , cachingAccesses=23730, cachingHits=23302, cachingHitsRatio=98.20%, evictions=9049, evicted=0, evictedPerRun=0.0 2022-03-06 04:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 04:46:54,542 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646541714541","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:37358","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 04:46:57,406 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646541117004","responsesize":"14879","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 9 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900402,"client":"172.30.41.118:37244","multi.service_calls":0,"queuetimems":2465,"class":"HRegionServer","multi.mutations":9} 2022-03-06 04:47:00,517 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646541120506","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 7 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900011,"client":"172.31.0.219:46018","multi.service_calls":0,"queuetimems":5967,"class":"HRegionServer","multi.mutations":7} 2022-03-06 04:47:01,070 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646541721070","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:37358","multi.service_calls":0,"queuetimems":986,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:47:22,846 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646541117956","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":924890,"client":"172.30.41.118:37244","multi.service_calls":0,"queuetimems":3417,"class":"HRegionServer","multi.mutations":7} 2022-03-06 04:47:22,846 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646541442844","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 8 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.118:37292","multi.service_calls":0,"queuetimems":22773,"class":"HRegionServer","multi.mutations":8} 2022-03-06 04:47:22,875 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646541442874","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:46040","multi.service_calls":0,"queuetimems":22803,"class":"HRegionServer","multi.mutations":4} 2022-03-06 04:47:24,136 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646541744130","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300006,"client":"172.30.41.118:37358","multi.service_calls":0,"queuetimems":24047,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:47:24,137 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646541744124","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300013,"client":"172.30.41.118:37358","multi.service_calls":0,"queuetimems":24041,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:47:27,050 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646541747049","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:37358","multi.service_calls":0,"queuetimems":26965,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:47:33,155 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391. because 0a764e51258807790269e46c55b99391/e has an old edit so flush to free WALs after random delay 64856 ms 2022-03-06 04:49:43,055 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e. because 4333b489291b1f77671b8f2e78779a1e/__status_family___ has an old edit so flush to free WALs after random delay 172317 ms 2022-03-06 04:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=32533, hits=24434, hitRatio=75.11%, , cachingAccesses=24553, cachingHits=24125, cachingHitsRatio=98.26%, evictions=9079, evicted=0, evictedPerRun=0.0 2022-03-06 04:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 04:51:54,543 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646542014543","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:37424","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:52:00,517 ERROR [MemStoreFlusher.0] regionserver.MemStoreFlusher: Cache flush failed for region aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb. org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync result after 300000 ms for txid=389134, WAL system stuck? at org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:174) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:800) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:640) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:617) at org.apache.hadoop.hbase.regionserver.HRegion.doSyncOfUnflushedWALChanges(HRegion.java:2927) at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2864) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2722) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2696) at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2569) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:617) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:586) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$1000(MemStoreFlusher.java:68) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:360) 2022-03-06 04:52:00,518 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 7d1b53259f1d8280a47f7704f98f2c9f 1/1 column families, dataSize=222.78 KB heapSize=382.07 KB 2022-03-06 04:52:00,846 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646541720845","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:46052","multi.service_calls":0,"queuetimems":6304,"class":"HRegionServer","multi.mutations":5} 2022-03-06 04:52:02,976 INFO [regionserver/cvp328:16201.Chore.2] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. because 7d1b53259f1d8280a47f7704f98f2c9f/e has an old edit so flush to free WALs after random delay 58847 ms 2022-03-06 04:52:03,487 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646542023486","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:37424","multi.service_calls":0,"queuetimems":7154,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:52:22,846 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646541742845","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:55554","multi.service_calls":0,"queuetimems":28304,"class":"HRegionServer","multi.mutations":2} 2022-03-06 04:52:22,846 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646541442844","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.155:55524","multi.service_calls":0,"queuetimems":22773,"class":"HRegionServer","multi.mutations":7} 2022-03-06 04:52:22,847 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646542042846","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:37426","multi.service_calls":0,"queuetimems":28273,"class":"HRegionServer","multi.mutations":2} 2022-03-06 04:52:22,953 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646541742952","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 8 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:37358","multi.service_calls":0,"queuetimems":28411,"class":"HRegionServer","multi.mutations":8} 2022-03-06 04:52:23,062 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646541743061","responsesize":"11572","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 7 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:46052","multi.service_calls":0,"queuetimems":28520,"class":"HRegionServer","multi.mutations":7} 2022-03-06 04:52:24,137 ERROR [MemStoreFlusher.1] regionserver.MemStoreFlusher: Cache flush failed for region aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df. org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync result after 300000 ms for txid=389158, WAL system stuck? at org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:174) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:800) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:640) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:617) at org.apache.hadoop.hbase.regionserver.HRegion.doSyncOfUnflushedWALChanges(HRegion.java:2927) at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2864) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2722) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2696) at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2569) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:617) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:586) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$1000(MemStoreFlusher.java:68) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:360) 2022-03-06 04:52:24,137 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing ccdacf4b414cb8d1bc548efc77c663cb 1/1 column families, dataSize=119.72 KB heapSize=208.27 KB 2022-03-06 04:53:04,661 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1039ms No GCs detected 2022-03-06 04:53:20,923 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4486ms No GCs detected 2022-03-06 04:53:27,429 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1046ms No GCs detected 2022-03-06 04:53:38,228 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 04:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=33298, hits=25199, hitRatio=75.68%, , cachingAccesses=25318, cachingHits=24890, cachingHitsRatio=98.31%, evictions=9109, evicted=0, evictedPerRun=0.0 2022-03-06 04:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 04:56:54,545 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646542314544","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:55624","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:56:57,408 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646542017406","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.31.0.219:46072","multi.service_calls":0,"queuetimems":2863,"class":"HRegionServer","multi.mutations":2} 2022-03-06 04:57:00,085 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646541720083","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 6 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.155:55554","multi.service_calls":0,"queuetimems":5542,"class":"HRegionServer","multi.mutations":6} 2022-03-06 04:57:00,844 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646542020517","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 4 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600326,"client":"172.31.0.219:46072","multi.service_calls":0,"queuetimems":5974,"class":"HRegionServer","multi.mutations":4} 2022-03-06 04:57:00,848 WARN [MemStoreFlusher.1] regionserver.DefaultMemStore: Snapshot called again without clearing previous. Doing nothing. Another ongoing flush or did we fail last attempt? 2022-03-06 04:57:03,487 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646542323487","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:55624","multi.service_calls":0,"queuetimems":2640,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:57:18,012 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2335ms No GCs detected 2022-03-06 04:57:22,847 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646542342846","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:55624","multi.service_calls":0,"queuetimems":21999,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:57:22,847 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646542042846","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 8 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:37424","multi.service_calls":0,"queuetimems":28303,"class":"HRegionServer","multi.mutations":8} 2022-03-06 04:57:22,847 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646542342846","responsesize":"6613","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:55624","multi.service_calls":0,"queuetimems":28302,"class":"HRegionServer","multi.mutations":4} 2022-03-06 04:57:23,063 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646542343062","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:55624","multi.service_calls":0,"queuetimems":22215,"class":"HRegionServer","multi.mutations":1} 2022-03-06 04:57:24,137 ERROR [MemStoreFlusher.0] regionserver.MemStoreFlusher: Cache flush failed for region aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync result after 300000 ms for txid=389190, WAL system stuck? at org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:174) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:800) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:640) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:617) at org.apache.hadoop.hbase.regionserver.HRegion.doSyncOfUnflushedWALChanges(HRegion.java:2927) at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2864) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2722) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2696) at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2569) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:617) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:586) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$1000(MemStoreFlusher.java:68) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:360) 2022-03-06 04:57:24,138 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646542042875","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 9 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":601262,"client":"172.31.0.219:46072","multi.service_calls":0,"queuetimems":25467,"class":"HRegionServer","multi.mutations":9} 2022-03-06 04:57:24,138 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646542044137","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.31.0.219:46072","multi.service_calls":0,"queuetimems":26729,"class":"HRegionServer","multi.mutations":2} 2022-03-06 04:57:24,138 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 80a1ffe1dc2d8f258a6367d204dc64df 1/1 column families, dataSize=112.80 KB heapSize=210.56 KB 2022-03-06 04:57:24,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646542044137","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 6 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:37424","multi.service_calls":0,"queuetimems":26728,"class":"HRegionServer","multi.mutations":6} 2022-03-06 04:57:24,138 WARN [MemStoreFlusher.0] regionserver.DefaultMemStore: Snapshot called again without clearing previous. Doing nothing. Another ongoing flush or did we fail last attempt? 2022-03-06 04:57:27,051 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646542047050","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":600001,"client":"172.30.41.118:37424","multi.service_calls":0,"queuetimems":29641,"class":"HRegionServer","multi.mutations":4} 2022-03-06 04:57:59,657 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2137ms No GCs detected 2022-03-06 04:58:11,405 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 04:58:12,345 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 523ms. 2022-03-06 05:00:58,504 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 5909ms No GCs detected 2022-03-06 05:01:20,306 WARN [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 16830ms No GCs detected 2022-03-06 05:01:27,436 INFO [regionserver/cvp328:16201.Chore.1] hbase.ScheduledChore: Chore: CompactionChecker missed its start time 2022-03-06 05:01:27,436 INFO [regionserver/cvp328:16201.Chore.1] hbase.ScheduledChore: Chore: MemstoreFlusherChore missed its start time 2022-03-06 05:01:27,446 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 05:01:27,496 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 5487ms No GCs detected 2022-03-06 05:01:27,656 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 810ms. 2022-03-06 05:01:42,065 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4700ms No GCs detected 2022-03-06 05:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=34006, hits=25907, hitRatio=76.18%, , cachingAccesses=26026, cachingHits=25598, cachingHitsRatio=98.36%, evictions=9138, evicted=0, evictedPerRun=0.0 2022-03-06 05:01:48,050 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 05:01:51,731 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1939ms No GCs detected 2022-03-06 05:01:54,547 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646542614546","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:37580","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-06 05:01:57,410 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646542617408","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.155:55648","multi.service_calls":0,"queuetimems":2862,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:02:00,844 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646542620844","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:55648","multi.service_calls":0,"queuetimems":5701,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:02:00,848 ERROR [MemStoreFlusher.1] regionserver.MemStoreFlusher: Cache flush failed for region aeris_v2,\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb. org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync result after 300000 ms for txid=389208, WAL system stuck? at org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:174) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:800) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:640) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:617) at org.apache.hadoop.hbase.regionserver.HRegion.doSyncOfUnflushedWALChanges(HRegion.java:2927) at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2864) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2722) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2696) at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2569) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:617) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:586) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$1000(MemStoreFlusher.java:68) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:360) 2022-03-06 05:02:00,848 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 0a764e51258807790269e46c55b99391 1/1 column families, dataSize=287.20 KB heapSize=515.48 KB 2022-03-06 05:02:03,488 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646542623487","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:55648","multi.service_calls":0,"queuetimems":7168,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:02:07,436 INFO [regionserver/cvp328:16201.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391. because 0a764e51258807790269e46c55b99391/e has an old edit so flush to free WALs after random delay 142599 ms 2022-03-06 05:02:15,375 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2761ms No GCs detected 2022-03-06 05:02:22,847 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646542642847","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:37580","multi.service_calls":0,"queuetimems":28301,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:02:22,847 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646542642847","responsesize":"6613","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:37580","multi.service_calls":0,"queuetimems":28301,"class":"HRegionServer","multi.mutations":4} 2022-03-06 05:02:22,851 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646542642847","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300003,"client":"172.30.41.118:37580","multi.service_calls":0,"queuetimems":26193,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:02:22,954 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646542342953","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:46090","multi.service_calls":0,"queuetimems":28408,"class":"HRegionServer","multi.mutations":2} 2022-03-06 05:02:23,072 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646542643063","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300009,"client":"172.30.41.118:37580","multi.service_calls":0,"queuetimems":28517,"class":"HRegionServer","multi.mutations":2} 2022-03-06 05:02:24,138 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646542644138","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:55648","multi.service_calls":0,"queuetimems":26722,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:02:24,139 ERROR [MemStoreFlusher.0] regionserver.MemStoreFlusher: Cache flush failed for region aeris_v2,\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df. org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync result after 300000 ms for txid=389228, WAL system stuck? at org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:174) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:800) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:640) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:617) at org.apache.hadoop.hbase.regionserver.HRegion.doSyncOfUnflushedWALChanges(HRegion.java:2927) at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2864) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2722) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2696) at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2569) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:617) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:586) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$1000(MemStoreFlusher.java:68) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:360) 2022-03-06 05:02:24,139 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646542644138","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:55648","multi.service_calls":0,"queuetimems":29591,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:02:24,139 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 4333b489291b1f77671b8f2e78779a1e 1/1 column families, dataSize=2.25 KB heapSize=3.77 KB 2022-03-06 05:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=34781, hits=26682, hitRatio=76.71%, , cachingAccesses=26801, cachingHits=26373, cachingHitsRatio=98.40%, evictions=9168, evicted=0, evictedPerRun=0.0 2022-03-06 05:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 05:06:57,489 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646542917411","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300078,"client":"172.30.41.118:37638","multi.service_calls":0,"queuetimems":2833,"class":"HRegionServer","multi.mutations":2} 2022-03-06 05:07:03,064 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 05:07:03,068 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646542620085","responsesize":"11572","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 7 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":602983,"client":"172.31.0.219:46108","multi.service_calls":0,"queuetimems":5538,"class":"HRegionServer","multi.mutations":7} 2022-03-06 05:07:03,157 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2427ms No GCs detected 2022-03-06 05:07:03,157 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646542920844","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":302313,"client":"172.30.41.155:55662","multi.service_calls":0,"queuetimems":5929,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:07:03,166 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 470ms. 2022-03-06 05:07:03,673 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646542923673","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:55662","multi.service_calls":0,"queuetimems":8758,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:07:22,848 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646542942847","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300001,"client":"172.30.41.118:37640","multi.service_calls":0,"queuetimems":28299,"class":"HRegionServer","multi.mutations":3} 2022-03-06 05:07:22,955 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646542942954","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:55662","multi.service_calls":0,"queuetimems":28039,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:07:24,385 ERROR [MemStoreFlusher.0] regionserver.MemStoreFlusher: Cache flush failed for region __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e. org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync result after 300000 ms for txid=389264, WAL system stuck? at org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:174) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:800) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:640) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:617) at org.apache.hadoop.hbase.regionserver.HRegion.doSyncOfUnflushedWALChanges(HRegion.java:2927) at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2864) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2722) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2696) at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2569) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:617) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:586) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$1000(MemStoreFlusher.java:68) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:360) 2022-03-06 05:07:24,385 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 7d1b53259f1d8280a47f7704f98f2c9f 1/1 column families, dataSize=222.78 KB heapSize=382.30 KB 2022-03-06 05:07:24,386 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646542944138","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300248,"client":"172.30.41.118:37640","multi.service_calls":0,"queuetimems":26727,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:07:33,687 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 05:07:59,763 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 6884ms No GCs detected 2022-03-06 05:08:09,108 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3145ms No GCs detected 2022-03-06 05:08:34,458 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 05:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=35558, hits=27459, hitRatio=77.22%, , cachingAccesses=27578, cachingHits=27150, cachingHitsRatio=98.45%, evictions=9197, evicted=0, evictedPerRun=0.0 2022-03-06 05:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 05:11:54,652 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646542914548","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600104,"client":"172.30.41.118:37640","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-06 05:12:03,068 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646543223068","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:37714","multi.service_calls":0,"queuetimems":5577,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:12:03,069 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646542320847","responsesize":"23142","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 14 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1202222,"client":"172.31.0.219:46090","multi.service_calls":0,"queuetimems":6302,"class":"HRegionServer","multi.mutations":14} 2022-03-06 05:12:03,085 ERROR [MemStoreFlusher.1] regionserver.MemStoreFlusher: Cache flush failed for region aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391. org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync result after 300000 ms for txid=389274, WAL system stuck? at org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:174) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:800) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:640) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:617) at org.apache.hadoop.hbase.regionserver.HRegion.doSyncOfUnflushedWALChanges(HRegion.java:2927) at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2864) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2722) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2696) at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2569) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:617) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:586) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$1000(MemStoreFlusher.java:68) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:360) 2022-03-06 05:12:03,086 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 0a764e51258807790269e46c55b99391 1/1 column families, dataSize=287.20 KB heapSize=515.71 KB 2022-03-06 05:12:03,086 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646543217492","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":305594,"client":"172.30.41.118:37714","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:12:03,158 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646543223157","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:37714","multi.service_calls":0,"queuetimems":5666,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:12:03,674 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646543223674","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:37714","multi.service_calls":0,"queuetimems":6183,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:12:22,849 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646542942847","responsesize":"18180","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 11 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":600002,"client":"172.31.0.219:46126","multi.service_calls":0,"queuetimems":28299,"class":"HRegionServer","multi.mutations":11} 2022-03-06 05:12:22,850 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646542342847","responsesize":"24794","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 15 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.118:37506","multi.service_calls":0,"queuetimems":28302,"class":"HRegionServer","multi.mutations":15} 2022-03-06 05:12:22,850 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646543242849","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:37714","multi.service_calls":0,"queuetimems":25358,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:12:22,851 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646542942851","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:55662","multi.service_calls":0,"queuetimems":27936,"class":"HRegionServer","multi.mutations":2} 2022-03-06 05:12:22,955 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646543242955","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:37714","multi.service_calls":0,"queuetimems":25464,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:12:24,374 WARN [MemStoreFlusher.0] regionserver.DefaultMemStore: Snapshot called again without clearing previous. Doing nothing. Another ongoing flush or did we fail last attempt? 2022-03-06 05:12:24,374 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646542944139","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600235,"client":"172.30.41.118:37640","multi.service_calls":0,"queuetimems":26728,"class":"HRegionServer","multi.mutations":2} 2022-03-06 05:12:24,386 WARN [MemStoreFlusher.1] regionserver.DefaultMemStore: Snapshot called again without clearing previous. Doing nothing. Another ongoing flush or did we fail last attempt? 2022-03-06 05:12:24,386 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646542644138","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 7 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900248,"client":"172.31.0.219:46108","multi.service_calls":0,"queuetimems":29591,"class":"HRegionServer","multi.mutations":7} 2022-03-06 05:12:24,386 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646543244386","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:37714","multi.service_calls":0,"queuetimems":26895,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=36265, hits=28166, hitRatio=77.67%, , cachingAccesses=28285, cachingHits=27857, cachingHitsRatio=98.49%, evictions=9227, evicted=0, evictedPerRun=0.0 2022-03-06 05:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 05:16:54,653 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646543514653","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:37778","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:17:03,070 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646543523070","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:37778","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:17:03,676 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646543523674","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300002,"client":"172.30.41.118:37782","multi.service_calls":0,"queuetimems":531,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:17:22,956 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646543542955","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:37778","multi.service_calls":0,"queuetimems":19885,"class":"HRegionServer","multi.mutations":3} 2022-03-06 05:17:23,192 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646542943072","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900120,"client":"172.30.41.155:55662","multi.service_calls":0,"queuetimems":28156,"class":"HRegionServer","multi.mutations":3} 2022-03-06 05:17:24,374 ERROR [MemStoreFlusher.0] regionserver.MemStoreFlusher: Cache flush failed for region aeris_v2,\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f. org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync result after 300000 ms for txid=389316, WAL system stuck? at org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:174) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:800) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:640) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:617) at org.apache.hadoop.hbase.regionserver.HRegion.doSyncOfUnflushedWALChanges(HRegion.java:2927) at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2864) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2722) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2696) at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2569) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:617) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:586) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$1000(MemStoreFlusher.java:68) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:360) 2022-03-06 05:17:24,375 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646543523069","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":321305,"client":"172.31.0.219:46178","multi.service_calls":0,"queuetimems":8416,"class":"HRegionServer","multi.mutations":2} 2022-03-06 05:17:24,375 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646543523086","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":321288,"client":"172.30.41.118:37778","multi.service_calls":0,"queuetimems":16,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:17:24,375 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646543523158","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":321217,"client":"172.30.41.118:37778","multi.service_calls":0,"queuetimems":88,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:17:24,386 ERROR [MemStoreFlusher.1] regionserver.MemStoreFlusher: Cache flush failed for region aeris_v2,\x08,1646451286050.0a764e51258807790269e46c55b99391. org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync result after 300000 ms for txid=389326, WAL system stuck? at org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:174) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:800) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:640) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:617) at org.apache.hadoop.hbase.regionserver.HRegion.doSyncOfUnflushedWALChanges(HRegion.java:2927) at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2864) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2722) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2696) at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2569) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:617) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:586) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$1000(MemStoreFlusher.java:68) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:360) 2022-03-06 05:17:24,386 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646543542852","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":301534,"client":"172.30.41.155:55696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:17:24,387 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646543544386","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:37778","multi.service_calls":0,"queuetimems":21316,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:17:24,387 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646543544374","responsesize":"11569","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300012,"client":"172.31.0.219:46178","multi.service_calls":0,"queuetimems":21304,"class":"HRegionServer","multi.mutations":7} 2022-03-06 05:17:27,066 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646542647051","responsesize":"13230","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 8 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200015,"client":"172.30.41.155:55648","multi.service_calls":0,"queuetimems":29635,"class":"HRegionServer","multi.mutations":8} 2022-03-06 05:20:15,208 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1550ms No GCs detected 2022-03-06 05:20:29,256 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1054ms No GCs detected 2022-03-06 05:20:33,543 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1168ms No GCs detected 2022-03-06 05:20:38,632 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 05:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=37091, hits=28992, hitRatio=78.16%, , cachingAccesses=29111, cachingHits=28683, cachingHitsRatio=98.53%, evictions=9257, evicted=0, evictedPerRun=0.0 2022-03-06 05:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 05:21:54,655 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646543814654","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:55708","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:22:22,850 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646543542850","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 4 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.31.0.219:46178","multi.service_calls":0,"queuetimems":19780,"class":"HRegionServer","multi.mutations":4} 2022-03-06 05:22:23,193 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646543843192","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:46220","multi.service_calls":0,"queuetimems":28537,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:22:24,375 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646543844375","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:46220","multi.service_calls":0,"queuetimems":29720,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:22:24,375 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646543844375","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:46220","multi.service_calls":0,"queuetimems":29720,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:22:24,375 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646543844375","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:55708","multi.service_calls":0,"queuetimems":29720,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:22:24,387 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646543844387","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:46220","multi.service_calls":0,"queuetimems":29732,"class":"HRegionServer","multi.mutations":2} 2022-03-06 05:22:27,066 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646543847066","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:55708","multi.service_calls":0,"queuetimems":29701,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=37820, hits=29721, hitRatio=78.59%, , cachingAccesses=29840, cachingHits=29412, cachingHitsRatio=98.57%, evictions=9287, evicted=0, evictedPerRun=0.0 2022-03-06 05:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 05:27:03,072 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646543823071","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:37842","multi.service_calls":0,"queuetimems":8417,"class":"HRegionServer","multi.mutations":3} 2022-03-06 05:27:03,678 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646543823676","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.155:55708","multi.service_calls":0,"queuetimems":9021,"class":"HRegionServer","multi.mutations":4} 2022-03-06 05:27:22,850 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646543542849","responsesize":"18183","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 11 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.30.41.118:37778","multi.service_calls":0,"queuetimems":28196,"class":"HRegionServer","multi.mutations":11} 2022-03-06 05:27:22,851 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646544142851","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:37914","multi.service_calls":0,"queuetimems":28195,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:27:22,957 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646543842956","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:37842","multi.service_calls":0,"queuetimems":28301,"class":"HRegionServer","multi.mutations":6} 2022-03-06 05:27:24,376 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646543542850","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 4 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":901525,"client":"172.31.0.219:46178","multi.service_calls":0,"queuetimems":28197,"class":"HRegionServer","multi.mutations":4} 2022-03-06 05:27:24,376 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646544144376","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:46260","multi.service_calls":0,"queuetimems":29720,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:27:24,387 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646543844386","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 6 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:46220","multi.service_calls":0,"queuetimems":29731,"class":"HRegionServer","multi.mutations":6} 2022-03-06 05:27:24,387 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646543844387","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:37842","multi.service_calls":0,"queuetimems":21316,"class":"HRegionServer","multi.mutations":2} 2022-03-06 05:27:27,067 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646544147067","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:37914","multi.service_calls":0,"queuetimems":2691,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:27:33,083 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646543553082","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 4 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:37788","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-06 05:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=38575, hits=30476, hitRatio=79.00%, , cachingAccesses=30595, cachingHits=30167, cachingHitsRatio=98.60%, evictions=9317, evicted=0, evictedPerRun=0.0 2022-03-06 05:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-06 05:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 05:31:54,657 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646544114656","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:55724","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-06 05:32:03,138 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646544423072","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300065,"client":"172.30.41.155:55740","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:32:22,852 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646544442851","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:37968","multi.service_calls":0,"queuetimems":19779,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:32:24,376 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646544444376","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:37968","multi.service_calls":0,"queuetimems":16839,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:32:24,376 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646544444376","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:46298","multi.service_calls":0,"queuetimems":21303,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:32:24,376 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646544144376","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 9 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.31.0.219:46260","multi.service_calls":0,"queuetimems":29720,"class":"HRegionServer","multi.mutations":9} 2022-03-06 05:32:24,377 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646544144376","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 9 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:37914","multi.service_calls":0,"queuetimems":29720,"class":"HRegionServer","multi.mutations":9} 2022-03-06 05:32:24,388 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646544444387","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.30.41.118:37920","multi.service_calls":0,"queuetimems":19422,"class":"HRegionServer","multi.mutations":2} 2022-03-06 05:32:24,388 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646544444387","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:55740","multi.service_calls":0,"queuetimems":16784,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:32:27,068 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646544447067","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:55740","multi.service_calls":0,"queuetimems":20135,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:32:33,083 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646544453083","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:37968","multi.service_calls":0,"queuetimems":26043,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:33:57,591 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1635ms No GCs detected 2022-03-06 05:34:39,760 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 5500ms No GCs detected 2022-03-06 05:35:13,936 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 05:35:18,732 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1070ms No GCs detected 2022-03-06 05:35:23,907 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2060ms No GCs detected 2022-03-06 05:35:44,875 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 05:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=39327, hits=31228, hitRatio=79.41%, , cachingAccesses=31347, cachingHits=30919, cachingHitsRatio=98.63%, evictions=9347, evicted=0, evictedPerRun=0.0 2022-03-06 05:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 05:37:22,852 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646544742852","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:46324","multi.service_calls":0,"queuetimems":28193,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:37:22,958 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646544442957","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:46298","multi.service_calls":0,"queuetimems":19884,"class":"HRegionServer","multi.mutations":3} 2022-03-06 05:37:24,377 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646544744376","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:38030","multi.service_calls":0,"queuetimems":21237,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:37:24,377 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646544744377","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:55758","multi.service_calls":0,"queuetimems":29718,"class":"HRegionServer","multi.mutations":2} 2022-03-06 05:37:24,388 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646544744388","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:55758","multi.service_calls":0,"queuetimems":21249,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:37:24,388 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646544144387","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 8 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:37914","multi.service_calls":0,"queuetimems":11,"class":"HRegionServer","multi.mutations":8} 2022-03-06 05:37:24,388 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646544744388","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:38030","multi.service_calls":0,"queuetimems":21249,"class":"HRegionServer","multi.mutations":2} 2022-03-06 05:39:09,313 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1460ms No GCs detected 2022-03-06 05:39:16,059 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 05:39:16,466 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 1 failover attempts. Trying to failover after sleeping for 852ms. 2022-03-06 05:39:32,628 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1040ms No GCs detected 2022-03-06 05:39:34,837 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1708ms No GCs detected 2022-03-06 05:40:09,642 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2245ms No GCs detected 2022-03-06 05:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=40181, hits=32082, hitRatio=79.84%, , cachingAccesses=32201, cachingHits=31773, cachingHitsRatio=98.67%, evictions=9377, evicted=0, evictedPerRun=0.0 2022-03-06 05:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 05:41:54,659 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646544714658","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:46324","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 05:42:03,680 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646544423678","responsesize":"19835","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 12 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900002,"client":"172.30.41.118:37968","multi.service_calls":0,"queuetimems":605,"class":"HRegionServer","multi.mutations":12} 2022-03-06 05:42:22,852 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646544442851","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:55740","multi.service_calls":0,"queuetimems":19779,"class":"HRegionServer","multi.mutations":7} 2022-03-06 05:42:22,853 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646545042853","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.31.0.219:46302","multi.service_calls":0,"queuetimems":12772,"class":"HRegionServer","multi.mutations":2} 2022-03-06 05:42:23,195 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646544143193","responsesize":"9926","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 6 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200002,"client":"172.30.41.155:55724","multi.service_calls":0,"queuetimems":28537,"class":"HRegionServer","multi.mutations":6} 2022-03-06 05:42:24,377 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646545044377","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:46344","multi.service_calls":0,"queuetimems":1524,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:42:24,377 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646544744376","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:55758","multi.service_calls":0,"queuetimems":29718,"class":"HRegionServer","multi.mutations":4} 2022-03-06 05:42:24,389 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646545044388","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:38110","multi.service_calls":0,"queuetimems":10,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:42:24,389 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646545044388","responsesize":"8265","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:46344","multi.service_calls":0,"queuetimems":1534,"class":"HRegionServer","multi.mutations":5} 2022-03-06 05:42:33,084 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646544753083","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 5 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:55758","multi.service_calls":0,"queuetimems":29944,"class":"HRegionServer","multi.mutations":5} 2022-03-06 05:44:23,842 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3463ms No GCs detected 2022-03-06 05:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=40961, hits=32862, hitRatio=80.23%, , cachingAccesses=32981, cachingHits=32553, cachingHitsRatio=98.70%, evictions=9407, evicted=0, evictedPerRun=0.0 2022-03-06 05:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 05:46:54,661 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646545314660","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:38172","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:47:03,139 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646544723138","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:38030","multi.service_calls":0,"queuetimems":8480,"class":"HRegionServer","multi.mutations":3} 2022-03-06 05:47:03,682 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646545323681","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:38172","multi.service_calls":0,"queuetimems":9020,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:47:22,853 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646545342853","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:38172","multi.service_calls":0,"queuetimems":28193,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:47:22,853 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646545342853","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:55798","multi.service_calls":0,"queuetimems":28193,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:47:22,959 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646545042958","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:38110","multi.service_calls":0,"queuetimems":105,"class":"HRegionServer","multi.mutations":3} 2022-03-06 05:47:23,195 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646545343195","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:46376","multi.service_calls":0,"queuetimems":28534,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:47:24,378 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646545344377","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:38172","multi.service_calls":0,"queuetimems":29716,"class":"HRegionServer","multi.mutations":3} 2022-03-06 05:47:24,378 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646545044378","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":600000,"client":"172.30.41.118:38110","multi.service_calls":0,"queuetimems":1525,"class":"HRegionServer","multi.mutations":3} 2022-03-06 05:47:24,389 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646545344389","responsesize":"6613","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:38172","multi.service_calls":0,"queuetimems":29728,"class":"HRegionServer","multi.mutations":4} 2022-03-06 05:47:27,070 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646544747068","responsesize":"21487","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 13 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.31.0.219:46324","multi.service_calls":0,"queuetimems":23929,"class":"HRegionServer","multi.mutations":13} 2022-03-06 05:47:33,085 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646545353084","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:55798","multi.service_calls":0,"queuetimems":29403,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:48:54,924 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 05:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=41686, hits=33587, hitRatio=80.57%, , cachingAccesses=33706, cachingHits=33278, cachingHitsRatio=98.73%, evictions=9437, evicted=0, evictedPerRun=0.0 2022-03-06 05:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 05:51:54,663 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646545614662","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:38242","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:52:03,141 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646545623140","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:55818","multi.service_calls":0,"queuetimems":8478,"class":"HRegionServer","multi.mutations":2} 2022-03-06 05:52:03,683 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646545623682","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:55818","multi.service_calls":0,"queuetimems":9020,"class":"HRegionServer","multi.mutations":2} 2022-03-06 05:52:22,854 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646545642854","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.31.0.219:46416","multi.service_calls":0,"queuetimems":28162,"class":"HRegionServer","multi.mutations":2} 2022-03-06 05:52:22,960 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646545642959","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:46414","multi.service_calls":0,"queuetimems":19818,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:52:23,196 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646545643195","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:38242","multi.service_calls":0,"queuetimems":20054,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:52:24,378 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646544744377","responsesize":"21490","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 13 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200001,"client":"172.30.41.118:38030","multi.service_calls":0,"queuetimems":29718,"class":"HRegionServer","multi.mutations":13} 2022-03-06 05:52:24,389 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646545044388","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 7 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:46344","multi.service_calls":0,"queuetimems":10,"class":"HRegionServer","multi.mutations":7} 2022-03-06 05:52:24,389 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646545644389","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:46414","multi.service_calls":0,"queuetimems":21248,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:52:33,086 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646545653085","responsesize":"11569","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:46414","multi.service_calls":0,"queuetimems":38423,"class":"HRegionServer","multi.mutations":7} 2022-03-06 05:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=42460, hits=34361, hitRatio=80.93%, , cachingAccesses=34480, cachingHits=34052, cachingHitsRatio=98.76%, evictions=9467, evicted=0, evictedPerRun=0.0 2022-03-06 05:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 05:56:54,664 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646545914664","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:38312","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:57:22,854 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646545642853","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:38242","multi.service_calls":0,"queuetimems":28191,"class":"HRegionServer","multi.mutations":3} 2022-03-06 05:57:22,855 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646545942854","responsesize":"9917","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:38312","multi.service_calls":0,"queuetimems":28190,"class":"HRegionServer","multi.mutations":6} 2022-03-06 05:57:23,197 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646545943196","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:38312","multi.service_calls":0,"queuetimems":20054,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:57:24,379 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646545344377","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 6 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:55798","multi.service_calls":0,"queuetimems":29716,"class":"HRegionServer","multi.mutations":6} 2022-03-06 05:57:24,379 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646545644378","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 6 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:38242","multi.service_calls":0,"queuetimems":29716,"class":"HRegionServer","multi.mutations":6} 2022-03-06 05:57:24,389 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646545344389","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900000,"client":"172.30.41.118:38172","multi.service_calls":0,"queuetimems":29728,"class":"HRegionServer","multi.mutations":6} 2022-03-06 05:57:24,390 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646545944390","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:55852","multi.service_calls":0,"queuetimems":21248,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:57:24,390 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646545944390","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:55852","multi.service_calls":0,"queuetimems":21248,"class":"HRegionServer","multi.mutations":1} 2022-03-06 05:57:27,071 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646545647070","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.31.0.219:46414","multi.service_calls":0,"queuetimems":32408,"class":"HRegionServer","multi.mutations":2} 2022-03-06 05:57:33,087 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646545953086","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:55852","multi.service_calls":0,"queuetimems":29944,"class":"HRegionServer","multi.mutations":1} 2022-03-06 06:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=43210, hits=35111, hitRatio=81.26%, , cachingAccesses=35230, cachingHits=34802, cachingHitsRatio=98.79%, evictions=9497, evicted=0, evictedPerRun=0.0 2022-03-06 06:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 06:01:54,666 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646546214665","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:38380","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 06:02:03,142 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646545923141","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:55852","multi.service_calls":0,"queuetimems":8477,"class":"HRegionServer","multi.mutations":2} 2022-03-06 06:02:03,685 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646545923683","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.155:55852","multi.service_calls":0,"queuetimems":9019,"class":"HRegionServer","multi.mutations":3} 2022-03-06 06:02:22,855 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646546242854","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:38380","multi.service_calls":0,"queuetimems":28189,"class":"HRegionServer","multi.mutations":1} 2022-03-06 06:02:22,855 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646546242855","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:55888","multi.service_calls":0,"queuetimems":28190,"class":"HRegionServer","multi.mutations":1} 2022-03-06 06:02:23,197 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646546243197","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:38380","multi.service_calls":0,"queuetimems":28532,"class":"HRegionServer","multi.mutations":1} 2022-03-06 06:02:24,379 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646546244379","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:55888","multi.service_calls":0,"queuetimems":29714,"class":"HRegionServer","multi.mutations":1} 2022-03-06 06:02:24,390 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646546244390","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:55888","multi.service_calls":0,"queuetimems":29725,"class":"HRegionServer","multi.mutations":1} 2022-03-06 06:02:30,294 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646546250294","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:55896","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 06:02:31,720 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646546251720","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:38380","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 06:02:36,632 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646546256632","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:38380","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 06:05:33,765 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 06:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=43944, hits=35845, hitRatio=81.57%, , cachingAccesses=35964, cachingHits=35536, cachingHitsRatio=98.81%, evictions=9527, evicted=0, evictedPerRun=0.0 2022-03-06 06:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 06:07:03,143 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646546523143","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:55924","multi.service_calls":0,"queuetimems":8476,"class":"HRegionServer","multi.mutations":2} 2022-03-06 06:07:03,686 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646546523685","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:46470","multi.service_calls":0,"queuetimems":9018,"class":"HRegionServer","multi.mutations":3} 2022-03-06 06:07:22,961 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646545942960","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 6 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:46432","multi.service_calls":0,"queuetimems":28296,"class":"HRegionServer","multi.mutations":6} 2022-03-06 06:07:23,198 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646546543197","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.30.41.118:38448","multi.service_calls":0,"queuetimems":28500,"class":"HRegionServer","multi.mutations":2} 2022-03-06 06:07:24,380 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646545644378","responsesize":"9926","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 6 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200002,"client":"172.30.41.155:55818","multi.service_calls":0,"queuetimems":21237,"class":"HRegionServer","multi.mutations":6} 2022-03-06 06:07:24,380 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646545944378","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 7 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.31.0.219:46432","multi.service_calls":0,"queuetimems":29714,"class":"HRegionServer","multi.mutations":7} 2022-03-06 06:07:24,380 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646546244379","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 6 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:46454","multi.service_calls":0,"queuetimems":29713,"class":"HRegionServer","multi.mutations":6} 2022-03-06 06:07:24,390 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646546544390","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:38446","multi.service_calls":0,"queuetimems":21247,"class":"HRegionServer","multi.mutations":1} 2022-03-06 06:07:24,391 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646546244390","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:38380","multi.service_calls":0,"queuetimems":29725,"class":"HRegionServer","multi.mutations":3} 2022-03-06 06:07:30,328 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646546550327","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300001,"client":"172.30.41.118:38452","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 06:07:33,150 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646546553150","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:38460","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 06:07:36,634 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646546556633","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:55928","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 06:11:38,572 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1396ms No GCs detected 2022-03-06 06:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=44707, hits=36608, hitRatio=81.88%, , cachingAccesses=36727, cachingHits=36299, cachingHitsRatio=98.83%, evictions=9557, evicted=0, evictedPerRun=0.0 2022-03-06 06:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 06:12:03,687 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646546823687","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:55944","multi.service_calls":0,"queuetimems":542,"class":"HRegionServer","multi.mutations":1} 2022-03-06 06:12:22,856 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646546542855","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:38446","multi.service_calls":0,"queuetimems":28188,"class":"HRegionServer","multi.mutations":4} 2022-03-06 06:12:22,856 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646546542855","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:55924","multi.service_calls":0,"queuetimems":28188,"class":"HRegionServer","multi.mutations":3} 2022-03-06 06:12:24,391 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646546844391","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:46490","multi.service_calls":0,"queuetimems":1429,"class":"HRegionServer","multi.mutations":1} 2022-03-06 06:12:25,092 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646546544379","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 5 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600713,"client":"172.30.41.155:55924","multi.service_calls":0,"queuetimems":21236,"class":"HRegionServer","multi.mutations":5} 2022-03-06 06:12:25,092 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646546844380","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300712,"client":"172.30.41.155:55944","multi.service_calls":0,"queuetimems":1418,"class":"HRegionServer","multi.mutations":1} 2022-03-06 06:12:25,619 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646546844390","responsesize":"9917","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":301229,"client":"172.30.41.118:38520","multi.service_calls":0,"queuetimems":1428,"class":"HRegionServer","multi.mutations":6} 2022-03-06 06:12:33,151 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646546853151","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:38520","multi.service_calls":0,"queuetimems":10189,"class":"HRegionServer","multi.mutations":1} 2022-03-06 06:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=45468, hits=37369, hitRatio=82.19%, , cachingAccesses=37488, cachingHits=37060, cachingHitsRatio=98.86%, evictions=9587, evicted=0, evictedPerRun=0.0 2022-03-06 06:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 06:16:54,669 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646546514667","responsesize":"16531","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 10 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900002,"client":"172.30.41.118:38446","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":10} 2022-03-06 06:17:03,145 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646546823144","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 4 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:38520","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-06 06:17:03,688 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646547123688","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:38582","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 06:17:22,857 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646547142856","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:38582","multi.service_calls":0,"queuetimems":19168,"class":"HRegionServer","multi.mutations":1} 2022-03-06 06:17:22,857 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646547142856","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:55962","multi.service_calls":0,"queuetimems":19168,"class":"HRegionServer","multi.mutations":1} 2022-03-06 06:17:23,199 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646546843198","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 9 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:46490","multi.service_calls":0,"queuetimems":20053,"class":"HRegionServer","multi.mutations":9} 2022-03-06 06:17:24,392 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646547144391","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:55962","multi.service_calls":0,"queuetimems":20703,"class":"HRegionServer","multi.mutations":1} 2022-03-06 06:17:25,093 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646547145092","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:55962","multi.service_calls":0,"queuetimems":21404,"class":"HRegionServer","multi.mutations":1} 2022-03-06 06:17:25,093 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646547145093","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:55962","multi.service_calls":0,"queuetimems":21405,"class":"HRegionServer","multi.mutations":1} 2022-03-06 06:17:25,619 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646546844380","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 4 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":601239,"client":"172.31.0.219:46490","multi.service_calls":0,"queuetimems":21235,"class":"HRegionServer","multi.mutations":4} 2022-03-06 06:17:25,619 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646547145619","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:55962","multi.service_calls":0,"queuetimems":21931,"class":"HRegionServer","multi.mutations":1} 2022-03-06 06:17:30,329 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646546850328","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:46490","multi.service_calls":0,"queuetimems":27183,"class":"HRegionServer","multi.mutations":2} 2022-03-06 06:17:33,152 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646547153151","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:55962","multi.service_calls":0,"queuetimems":29463,"class":"HRegionServer","multi.mutations":1} 2022-03-06 06:17:36,635 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646546856634","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 4 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:55944","multi.service_calls":0,"queuetimems":13672,"class":"HRegionServer","multi.mutations":4} 2022-03-06 06:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=46208, hits=38109, hitRatio=82.47%, , cachingAccesses=38228, cachingHits=37800, cachingHitsRatio=98.88%, evictions=9617, evicted=0, evictedPerRun=0.0 2022-03-06 06:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 06:22:03,147 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646547423146","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:38664","multi.service_calls":0,"queuetimems":8476,"class":"HRegionServer","multi.mutations":3} 2022-03-06 06:22:03,689 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646547423688","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:55980","multi.service_calls":0,"queuetimems":542,"class":"HRegionServer","multi.mutations":1} 2022-03-06 06:22:22,857 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646547442857","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:38664","multi.service_calls":0,"queuetimems":28187,"class":"HRegionServer","multi.mutations":1} 2022-03-06 06:22:22,963 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646546842962","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 8 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:38520","multi.service_calls":0,"queuetimems":19818,"class":"HRegionServer","multi.mutations":8} 2022-03-06 06:22:23,201 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646547443199","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.31.0.219:46528","multi.service_calls":0,"queuetimems":28528,"class":"HRegionServer","multi.mutations":1} 2022-03-06 06:22:25,093 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646547445093","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:55980","multi.service_calls":0,"queuetimems":21947,"class":"HRegionServer","multi.mutations":1} 2022-03-06 06:22:25,094 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646547445093","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:38664","multi.service_calls":0,"queuetimems":21947,"class":"HRegionServer","multi.mutations":2} 2022-03-06 06:22:25,619 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646546844380","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 6 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":901239,"client":"172.30.41.155:55944","multi.service_calls":0,"queuetimems":21235,"class":"HRegionServer","multi.mutations":6} 2022-03-06 06:22:33,182 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646547453182","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:55982","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 06:22:36,686 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646547456686","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300000,"client":"172.30.41.118:38674","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 06:26:39,601 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 06:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=46894, hits=38795, hitRatio=82.73%, , cachingAccesses=38914, cachingHits=38486, cachingHitsRatio=98.90%, evictions=9647, evicted=0, evictedPerRun=0.0 2022-03-06 06:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 06:27:03,148 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646547723148","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:38734","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 06:27:03,690 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646547723689","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:55998","multi.service_calls":0,"queuetimems":541,"class":"HRegionServer","multi.mutations":1} 2022-03-06 06:27:18,388 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1093ms No GCs detected 2022-03-06 06:27:22,858 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646547442857","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:55980","multi.service_calls":0,"queuetimems":28187,"class":"HRegionServer","multi.mutations":3} 2022-03-06 06:27:22,964 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646547742963","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:55998","multi.service_calls":0,"queuetimems":105,"class":"HRegionServer","multi.mutations":1} 2022-03-06 06:27:23,201 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646547743201","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:38734","multi.service_calls":0,"queuetimems":343,"class":"HRegionServer","multi.mutations":1} 2022-03-06 06:27:24,393 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646547444392","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.31.0.219:46528","multi.service_calls":0,"queuetimems":29721,"class":"HRegionServer","multi.mutations":2} 2022-03-06 06:27:28,805 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1115ms No GCs detected 2022-03-06 06:27:34,740 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4434ms No GCs detected 2022-03-06 06:27:34,740 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646547453181","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":601558,"client":"172.30.41.155:55982","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 06:28:10,561 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 06:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=47705, hits=39606, hitRatio=83.02%, , cachingAccesses=39725, cachingHits=39297, cachingHitsRatio=98.92%, evictions=9677, evicted=0, evictedPerRun=0.0 2022-03-06 06:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-06 06:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 06:32:22,860 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646547742858","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.31.0.219:46546","multi.service_calls":0,"queuetimems":19702,"class":"HRegionServer","multi.mutations":2} 2022-03-06 06:32:23,202 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646548043202","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:46566","multi.service_calls":0,"queuetimems":343,"class":"HRegionServer","multi.mutations":1} 2022-03-06 06:32:24,393 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646548044393","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:46566","multi.service_calls":0,"queuetimems":1534,"class":"HRegionServer","multi.mutations":1} 2022-03-06 06:32:25,164 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646547745093","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 9 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600071,"client":"172.30.41.118:38734","multi.service_calls":0,"queuetimems":21945,"class":"HRegionServer","multi.mutations":9} 2022-03-06 06:32:25,621 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646547745620","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 4 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:46546","multi.service_calls":0,"queuetimems":22463,"class":"HRegionServer","multi.mutations":4} 2022-03-06 06:32:36,687 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646547756686","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:46546","multi.service_calls":0,"queuetimems":13827,"class":"HRegionServer","multi.mutations":2} 2022-03-06 06:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=48466, hits=40367, hitRatio=83.29%, , cachingAccesses=40486, cachingHits=40058, cachingHitsRatio=98.94%, evictions=9707, evicted=0, evictedPerRun=0.0 2022-03-06 06:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 06:36:54,672 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646547414670","responsesize":"11578","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200002,"client":"172.30.41.155:55980","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":7} 2022-03-06 06:37:03,150 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646548023149","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 5 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":600001,"client":"172.30.41.118:38806","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":5} 2022-03-06 06:37:03,691 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646548023690","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 9 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:38806","multi.service_calls":0,"queuetimems":541,"class":"HRegionServer","multi.mutations":9} 2022-03-06 06:37:22,860 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646548042858","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.155:56020","multi.service_calls":0,"queuetimems":19709,"class":"HRegionServer","multi.mutations":4} 2022-03-06 06:37:24,434 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646548344393","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300040,"client":"172.30.41.155:56056","multi.service_calls":0,"queuetimems":1529,"class":"HRegionServer","multi.mutations":2} 2022-03-06 06:37:25,621 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646547445619","responsesize":"21490","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 13 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.118:38664","multi.service_calls":0,"queuetimems":22472,"class":"HRegionServer","multi.mutations":13} 2022-03-06 06:37:25,633 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646548345632","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.155:56058","multi.service_calls":0,"queuetimems":2740,"class":"HRegionServer","multi.mutations":2} 2022-03-06 06:37:34,712 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646547453182","responsesize":"13230","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 8 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1201530,"client":"172.30.41.155:55982","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":8} 2022-03-06 06:37:34,740 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646547753183","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 8 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":901557,"client":"172.30.41.118:38734","multi.service_calls":0,"queuetimems":10325,"class":"HRegionServer","multi.mutations":8} 2022-03-06 06:37:36,688 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646548356687","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:46584","multi.service_calls":0,"queuetimems":11456,"class":"HRegionServer","multi.mutations":2} 2022-03-06 06:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=49247, hits=41148, hitRatio=83.55%, , cachingAccesses=41267, cachingHits=40839, cachingHitsRatio=98.96%, evictions=9737, evicted=0, evictedPerRun=0.0 2022-03-06 06:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 06:41:54,674 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646548614674","responsesize":"6613","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:38932","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":4} 2022-03-06 06:42:03,691 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646548623691","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:38932","multi.service_calls":0,"queuetimems":9017,"class":"HRegionServer","multi.mutations":1} 2022-03-06 06:42:22,865 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646548342861","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600004,"client":"172.30.41.118:38888","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 06:42:22,965 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646548042964","responsesize":"19835","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 12 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.31.0.219:46566","multi.service_calls":0,"queuetimems":19814,"class":"HRegionServer","multi.mutations":12} 2022-03-06 06:42:23,203 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646548343202","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 8 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:38888","multi.service_calls":0,"queuetimems":341,"class":"HRegionServer","multi.mutations":8} 2022-03-06 06:42:24,434 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646548644434","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:38932","multi.service_calls":0,"queuetimems":29760,"class":"HRegionServer","multi.mutations":3} 2022-03-06 06:42:25,164 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646547745094","responsesize":"16534","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 10 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200070,"client":"172.30.41.155:55998","multi.service_calls":0,"queuetimems":2236,"class":"HRegionServer","multi.mutations":10} 2022-03-06 06:42:25,622 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646548645621","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:46602","multi.service_calls":0,"queuetimems":30947,"class":"HRegionServer","multi.mutations":2} 2022-03-06 06:42:25,633 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646548645633","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:38932","multi.service_calls":0,"queuetimems":22482,"class":"HRegionServer","multi.mutations":1} 2022-03-06 06:42:34,713 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646548654713","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:38940","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 06:42:34,741 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646548654740","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:56096","multi.service_calls":0,"queuetimems":27,"class":"HRegionServer","multi.mutations":1} 2022-03-06 06:42:34,747 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646548054746","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 6 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:56020","multi.service_calls":0,"queuetimems":31597,"class":"HRegionServer","multi.mutations":6} 2022-03-06 06:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=50061, hits=41962, hitRatio=83.82%, , cachingAccesses=42081, cachingHits=41653, cachingHitsRatio=98.98%, evictions=9767, evicted=0, evictedPerRun=0.0 2022-03-06 06:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 06:47:03,152 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646548623151","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:56092","multi.service_calls":0,"queuetimems":8477,"class":"HRegionServer","multi.mutations":4} 2022-03-06 06:47:22,862 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646548642860","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.155:56092","multi.service_calls":0,"queuetimems":28186,"class":"HRegionServer","multi.mutations":3} 2022-03-06 06:47:23,204 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646548943203","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:38984","multi.service_calls":0,"queuetimems":19511,"class":"HRegionServer","multi.mutations":1} 2022-03-06 06:47:24,435 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646548944434","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:56130","multi.service_calls":0,"queuetimems":20742,"class":"HRegionServer","multi.mutations":1} 2022-03-06 06:47:25,165 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646548945164","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:56130","multi.service_calls":0,"queuetimems":21472,"class":"HRegionServer","multi.mutations":3} 2022-03-06 06:47:25,231 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646548345230","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 6 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:56056","multi.service_calls":0,"queuetimems":2366,"class":"HRegionServer","multi.mutations":6} 2022-03-06 06:47:34,714 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646548954714","responsesize":"9917","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:38984","multi.service_calls":0,"queuetimems":31022,"class":"HRegionServer","multi.mutations":6} 2022-03-06 06:47:34,741 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646548954741","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300000,"client":"172.30.41.118:38984","multi.service_calls":0,"queuetimems":22703,"class":"HRegionServer","multi.mutations":1} 2022-03-06 06:47:34,748 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646548954747","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300001,"client":"172.30.41.118:38986","multi.service_calls":0,"queuetimems":11514,"class":"HRegionServer","multi.mutations":1} 2022-03-06 06:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=50812, hits=42713, hitRatio=84.06%, , cachingAccesses=42832, cachingHits=42404, cachingHitsRatio=99.00%, evictions=9797, evicted=0, evictedPerRun=0.0 2022-03-06 06:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 06:51:54,676 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646548914675","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 5 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:38984","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":5} 2022-03-06 06:52:03,693 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646548923692","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:56130","multi.service_calls":0,"queuetimems":9017,"class":"HRegionServer","multi.mutations":3} 2022-03-06 06:52:22,966 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646548942966","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.31.0.219:46624","multi.service_calls":0,"queuetimems":28291,"class":"HRegionServer","multi.mutations":2} 2022-03-06 06:52:25,165 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646549245165","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:39036","multi.service_calls":0,"queuetimems":2302,"class":"HRegionServer","multi.mutations":1} 2022-03-06 06:52:25,623 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646548945622","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:46624","multi.service_calls":0,"queuetimems":21930,"class":"HRegionServer","multi.mutations":4} 2022-03-06 06:52:34,742 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646549254742","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:39036","multi.service_calls":0,"queuetimems":11879,"class":"HRegionServer","multi.mutations":1} 2022-03-06 06:52:34,748 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646549254748","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:46646","multi.service_calls":0,"queuetimems":11885,"class":"HRegionServer","multi.mutations":1} 2022-03-06 06:52:36,689 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646548656688","responsesize":"18183","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 11 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.30.41.118:38940","multi.service_calls":0,"queuetimems":1975,"class":"HRegionServer","multi.mutations":11} 2022-03-06 06:53:39,627 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1017ms No GCs detected 2022-03-06 06:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=51563, hits=43464, hitRatio=84.29%, , cachingAccesses=43583, cachingHits=43155, cachingHitsRatio=99.02%, evictions=9826, evicted=0, evictedPerRun=0.0 2022-03-06 06:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 06:56:54,678 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646549514678","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:39084","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 06:57:03,154 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646549223153","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 5 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:39036","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":5} 2022-03-06 06:57:03,694 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646549523693","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:46666","multi.service_calls":0,"queuetimems":9015,"class":"HRegionServer","multi.mutations":3} 2022-03-06 06:57:22,866 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646549242863","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 9 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.118:39036","multi.service_calls":0,"queuetimems":19710,"class":"HRegionServer","multi.mutations":9} 2022-03-06 06:57:22,866 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646548942865","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 6 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:38984","multi.service_calls":0,"queuetimems":28190,"class":"HRegionServer","multi.mutations":6} 2022-03-06 06:57:24,436 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646549244435","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 8 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:46646","multi.service_calls":0,"queuetimems":21282,"class":"HRegionServer","multi.mutations":8} 2022-03-06 06:57:25,624 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646549545623","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:46666","multi.service_calls":0,"queuetimems":21929,"class":"HRegionServer","multi.mutations":1} 2022-03-06 06:57:25,634 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646548945633","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 5 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:56130","multi.service_calls":0,"queuetimems":30958,"class":"HRegionServer","multi.mutations":5} 2022-03-06 06:57:34,743 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646549554743","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:39090","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 06:57:36,690 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646549556689","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:46668","multi.service_calls":0,"queuetimems":1946,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=52347, hits=44248, hitRatio=84.53%, , cachingAccesses=44367, cachingHits=43939, cachingHitsRatio=99.04%, evictions=9856, evicted=0, evictedPerRun=0.0 2022-03-06 07:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 07:01:54,680 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646549814680","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:39134","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":2} 2022-03-06 07:02:22,869 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646549842866","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300003,"client":"172.30.41.155:56250","multi.service_calls":0,"queuetimems":28187,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:02:22,869 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646549842866","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300003,"client":"172.30.41.155:56248","multi.service_calls":0,"queuetimems":28156,"class":"HRegionServer","multi.mutations":2} 2022-03-06 07:02:22,968 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646549542967","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 8 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:39084","multi.service_calls":0,"queuetimems":28289,"class":"HRegionServer","multi.mutations":8} 2022-03-06 07:02:23,206 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646549243204","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 6 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:56170","multi.service_calls":0,"queuetimems":20051,"class":"HRegionServer","multi.mutations":6} 2022-03-06 07:02:24,437 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646549844436","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:39134","multi.service_calls":0,"queuetimems":21281,"class":"HRegionServer","multi.mutations":2} 2022-03-06 07:02:25,166 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646549545165","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:39084","multi.service_calls":0,"queuetimems":21471,"class":"HRegionServer","multi.mutations":3} 2022-03-06 07:02:25,233 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646549245231","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:56170","multi.service_calls":0,"queuetimems":22078,"class":"HRegionServer","multi.mutations":3} 2022-03-06 07:02:25,635 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646549845634","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:39134","multi.service_calls":0,"queuetimems":22479,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:02:34,716 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646549254714","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 4 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:46646","multi.service_calls":0,"queuetimems":31561,"class":"HRegionServer","multi.mutations":4} 2022-03-06 07:02:34,744 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646549854744","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:39134","multi.service_calls":0,"queuetimems":31589,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:02:34,749 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646549554748","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 5 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:56210","multi.service_calls":0,"queuetimems":5,"class":"HRegionServer","multi.mutations":5} 2022-03-06 07:02:37,241 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646549856690","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300551,"client":"172.30.41.155:56250","multi.service_calls":0,"queuetimems":33535,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:03:54,419 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1319ms No GCs detected 2022-03-06 07:04:03,216 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1046ms No GCs detected 2022-03-06 07:04:05,909 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2192ms No GCs detected 2022-03-06 07:06:37,654 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1555ms No GCs detected 2022-03-06 07:06:44,611 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=52990, hits=44891, hitRatio=84.72%, , cachingAccesses=45010, cachingHits=44582, cachingHitsRatio=99.05%, evictions=9886, evicted=0, evictedPerRun=0.0 2022-03-06 07:06:45,850 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1690ms No GCs detected 2022-03-06 07:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 07:07:22,870 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646550142869","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:56284","multi.service_calls":0,"queuetimems":28188,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:07:22,870 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646550142869","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:56284","multi.service_calls":0,"queuetimems":28188,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:07:22,968 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646550142968","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:56284","multi.service_calls":0,"queuetimems":28287,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:07:23,206 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646550143206","responsesize":"11569","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:46706","multi.service_calls":0,"queuetimems":28524,"class":"HRegionServer","multi.mutations":7} 2022-03-06 07:07:24,713 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646550144712","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:39196","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 07:07:25,168 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646550145167","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:46714","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:07:34,716 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646550154716","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:56290","multi.service_calls":0,"queuetimems":10003,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:07:34,745 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646550154744","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:39196","multi.service_calls":0,"queuetimems":9510,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:07:34,749 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646550154749","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:39196","multi.service_calls":0,"queuetimems":9515,"class":"HRegionServer","multi.mutations":2} 2022-03-06 07:07:37,241 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646550157241","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.31.0.219:46712","multi.service_calls":0,"queuetimems":11977,"class":"HRegionServer","multi.mutations":2} 2022-03-06 07:08:26,342 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1051ms No GCs detected 2022-03-06 07:11:31,154 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2530ms No GCs detected 2022-03-06 07:11:33,520 INFO [regionserver/cvp328:16201.Chore.3] hbase.ScheduledChore: Chore: CompactionChecker missed its start time 2022-03-06 07:11:33,521 INFO [regionserver/cvp328:16201.Chore.3] hbase.ScheduledChore: Chore: MemstoreFlusherChore missed its start time 2022-03-06 07:11:33,521 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1866ms No GCs detected 2022-03-06 07:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=53717, hits=45618, hitRatio=84.92%, , cachingAccesses=45737, cachingHits=45309, cachingHitsRatio=99.06%, evictions=9916, evicted=0, evictedPerRun=0.0 2022-03-06 07:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 07:11:52,699 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 07:11:54,682 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646550114681","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:39186","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-06 07:12:03,156 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646549823154","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.155:56250","multi.service_calls":0,"queuetimems":8474,"class":"HRegionServer","multi.mutations":7} 2022-03-06 07:12:03,695 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646549823694","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 6 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:46686","multi.service_calls":0,"queuetimems":9014,"class":"HRegionServer","multi.mutations":6} 2022-03-06 07:12:23,207 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646550443206","responsesize":"6613","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:39240","multi.service_calls":0,"queuetimems":335,"class":"HRegionServer","multi.mutations":4} 2022-03-06 07:12:25,234 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646550145233","responsesize":"11572","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:39196","multi.service_calls":0,"queuetimems":521,"class":"HRegionServer","multi.mutations":7} 2022-03-06 07:12:25,636 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646550145635","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:56290","multi.service_calls":0,"queuetimems":922,"class":"HRegionServer","multi.mutations":3} 2022-03-06 07:12:34,126 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646550454125","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.31.0.219:46748","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 07:12:34,717 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646550454717","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:56304","multi.service_calls":0,"queuetimems":11846,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:12:34,717 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646550454717","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:39240","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:12:34,746 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646550454745","responsesize":"9917","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:39240","multi.service_calls":0,"queuetimems":28,"class":"HRegionServer","multi.mutations":6} 2022-03-06 07:12:44,841 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646550464841","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:46746","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:12:44,841 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646550464841","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:46746","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=54471, hits=46372, hitRatio=85.13%, , cachingAccesses=46491, cachingHits=46063, cachingHitsRatio=99.08%, evictions=9946, evicted=0, evictedPerRun=0.0 2022-03-06 07:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 07:16:54,684 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646550714683","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300001,"client":"172.30.41.118:39290","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:17:03,160 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646550723157","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300003,"client":"172.30.41.118:39290","multi.service_calls":0,"queuetimems":8474,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:17:03,696 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646550723695","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:39290","multi.service_calls":0,"queuetimems":9012,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:17:22,969 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646550442968","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 4 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:56304","multi.service_calls":0,"queuetimems":97,"class":"HRegionServer","multi.mutations":4} 2022-03-06 07:17:23,208 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646550743207","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:39290","multi.service_calls":0,"queuetimems":28524,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:17:25,236 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646550745234","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.118:39290","multi.service_calls":0,"queuetimems":30551,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:17:25,626 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646549845624","responsesize":"21490","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 13 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.118:39134","multi.service_calls":0,"queuetimems":22469,"class":"HRegionServer","multi.mutations":13} 2022-03-06 07:17:25,636 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646550745636","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:46774","multi.service_calls":0,"queuetimems":30952,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:17:34,718 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646550754718","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:46778","multi.service_calls":0,"queuetimems":591,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:17:34,746 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646550754746","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:56322","multi.service_calls":0,"queuetimems":28,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:17:44,842 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646550764841","responsesize":"6613","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:56322","multi.service_calls":0,"queuetimems":10123,"class":"HRegionServer","multi.mutations":4} 2022-03-06 07:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=55248, hits=47149, hitRatio=85.34%, , cachingAccesses=47268, cachingHits=46840, cachingHitsRatio=99.09%, evictions=9976, evicted=0, evictedPerRun=0.0 2022-03-06 07:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 07:22:22,873 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646550442871","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.118:39240","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-06 07:22:22,873 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646550442871","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 5 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.118:39240","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":5} 2022-03-06 07:22:25,626 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646551045626","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:56338","multi.service_calls":0,"queuetimems":22465,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:22:25,637 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646551045636","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:39362","multi.service_calls":0,"queuetimems":22475,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:22:34,128 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646550754127","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:39304","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 07:22:34,718 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646550754718","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:56322","multi.service_calls":0,"queuetimems":591,"class":"HRegionServer","multi.mutations":4} 2022-03-06 07:22:34,749 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646551054749","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:39374","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 07:22:34,769 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646551054769","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300000,"client":"172.30.41.118:39376","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:22:44,842 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646550764841","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:46778","multi.service_calls":0,"queuetimems":10123,"class":"HRegionServer","multi.mutations":3} 2022-03-06 07:22:56,746 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646551076745","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:56340","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 07:24:55,042 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3156ms No GCs detected 2022-03-06 07:24:55,426 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 07:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=55964, hits=47865, hitRatio=85.53%, , cachingAccesses=47984, cachingHits=47556, cachingHitsRatio=99.11%, evictions=10006, evicted=0, evictedPerRun=0.0 2022-03-06 07:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 07:26:54,686 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646551014685","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 8 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:39362","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":8} 2022-03-06 07:27:06,097 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1328ms No GCs detected 2022-03-06 07:27:22,912 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1002ms No GCs detected 2022-03-06 07:27:23,209 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646551043208","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.31.0.219:46794","multi.service_calls":0,"queuetimems":28523,"class":"HRegionServer","multi.mutations":3} 2022-03-06 07:27:22,970 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646551042969","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:39362","multi.service_calls":0,"queuetimems":28284,"class":"HRegionServer","multi.mutations":3} 2022-03-06 07:27:22,913 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646551342874","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300038,"client":"172.30.41.155:56360","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:27:25,237 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646551045236","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 8 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:46794","multi.service_calls":0,"queuetimems":30551,"class":"HRegionServer","multi.mutations":8} 2022-03-06 07:27:25,627 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646551345626","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:46818","multi.service_calls":0,"queuetimems":2752,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:27:27,918 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 07:27:34,129 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646551354128","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:39450","multi.service_calls":0,"queuetimems":8490,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:27:34,770 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646551354769","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.30.41.118:39452","multi.service_calls":0,"queuetimems":11865,"class":"HRegionServer","multi.mutations":2} 2022-03-06 07:27:44,843 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646551364842","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:56360","multi.service_calls":0,"queuetimems":19204,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:29:10,589 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1114ms No GCs detected 2022-03-06 07:29:27,944 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 07:30:49,427 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1183ms No GCs detected 2022-03-06 07:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=56642, hits=48543, hitRatio=85.70%, , cachingAccesses=48662, cachingHits=48234, cachingHitsRatio=99.12%, evictions=10036, evicted=0, evictedPerRun=0.0 2022-03-06 07:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-06 07:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 07:32:03,268 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646551023161","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900107,"client":"172.30.41.155:56338","multi.service_calls":0,"queuetimems":8476,"class":"HRegionServer","multi.mutations":3} 2022-03-06 07:32:04,452 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646551023696","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 7 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900756,"client":"172.30.41.155:56338","multi.service_calls":0,"queuetimems":9011,"class":"HRegionServer","multi.mutations":7} 2022-03-06 07:32:22,912 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646551342874","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":600038,"client":"172.30.41.118:39450","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 07:32:24,695 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646551644695","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:39526","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:32:25,238 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646551645238","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:56376","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:32:34,719 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646551354719","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 5 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:39450","multi.service_calls":0,"queuetimems":9081,"class":"HRegionServer","multi.mutations":5} 2022-03-06 07:32:34,771 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646551654770","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:46834","multi.service_calls":0,"queuetimems":40083,"class":"HRegionServer","multi.mutations":3} 2022-03-06 07:36:30,482 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 07:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=57337, hits=49238, hitRatio=85.87%, , cachingAccesses=49357, cachingHits=48929, cachingHitsRatio=99.13%, evictions=10066, evicted=0, evictedPerRun=0.0 2022-03-06 07:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 07:36:54,688 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646551614687","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 5 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:39512","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":5} 2022-03-06 07:37:22,913 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646551942913","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:56392","multi.service_calls":0,"queuetimems":19644,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:37:24,696 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646551644695","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 9 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:39526","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":9} 2022-03-06 07:37:25,238 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646551645237","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 5 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:56376","multi.service_calls":0,"queuetimems":30550,"class":"HRegionServer","multi.mutations":5} 2022-03-06 07:37:25,239 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646551945238","responsesize":"6613","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:46852","multi.service_calls":0,"queuetimems":21969,"class":"HRegionServer","multi.mutations":4} 2022-03-06 07:37:25,628 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646551645627","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 6 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:46834","multi.service_calls":0,"queuetimems":389,"class":"HRegionServer","multi.mutations":6} 2022-03-06 07:37:25,639 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646551345637","responsesize":"18183","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 11 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900002,"client":"172.30.41.118:39450","multi.service_calls":0,"queuetimems":2763,"class":"HRegionServer","multi.mutations":11} 2022-03-06 07:37:34,720 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646551954720","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:46852","multi.service_calls":0,"queuetimems":31451,"class":"HRegionServer","multi.mutations":2} 2022-03-06 07:37:34,751 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646551354749","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 7 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.31.0.219:46818","multi.service_calls":0,"queuetimems":11875,"class":"HRegionServer","multi.mutations":7} 2022-03-06 07:37:34,771 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646551954771","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:39572","multi.service_calls":0,"queuetimems":31329,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:37:44,917 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646551664843","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600074,"client":"172.30.41.155:56376","multi.service_calls":0,"queuetimems":19605,"class":"HRegionServer","multi.mutations":6} 2022-03-06 07:38:15,713 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2192ms No GCs detected 2022-03-06 07:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=58021, hits=49922, hitRatio=86.04%, , cachingAccesses=50041, cachingHits=49613, cachingHitsRatio=99.14%, evictions=10096, evicted=0, evictedPerRun=0.0 2022-03-06 07:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 07:42:04,454 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646551924452","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.155:56392","multi.service_calls":0,"queuetimems":1183,"class":"HRegionServer","multi.mutations":4} 2022-03-06 07:42:24,697 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646552244697","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:56414","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:42:25,239 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646552245239","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:56414","multi.service_calls":0,"queuetimems":542,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:42:25,239 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646552245239","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:56414","multi.service_calls":0,"queuetimems":542,"class":"HRegionServer","multi.mutations":3} 2022-03-06 07:42:25,629 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646552245628","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:39654","multi.service_calls":0,"queuetimems":901,"class":"HRegionServer","multi.mutations":2} 2022-03-06 07:42:34,130 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646551654129","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 8 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:46834","multi.service_calls":0,"queuetimems":8891,"class":"HRegionServer","multi.mutations":8} 2022-03-06 07:42:34,721 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646552254720","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:56414","multi.service_calls":0,"queuetimems":9481,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:42:34,751 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646552254751","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:39656","multi.service_calls":0,"queuetimems":9112,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:42:34,773 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646552254772","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:39656","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:42:56,696 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646552276696","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300000,"client":"172.30.41.118:39660","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:42:56,932 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646551376746","responsesize":"16534","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 10 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200186,"client":"172.30.41.155:56360","multi.service_calls":0,"queuetimems":31108,"class":"HRegionServer","multi.mutations":10} 2022-03-06 07:44:03,942 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 07:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=58752, hits=50653, hitRatio=86.21%, , cachingAccesses=50772, cachingHits=50344, cachingHitsRatio=99.16%, evictions=10126, evicted=0, evictedPerRun=0.0 2022-03-06 07:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 07:46:54,690 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646552214689","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:39646","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-06 07:47:03,270 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646551923269","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:39572","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-06 07:47:04,455 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646552524455","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:46888","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:47:22,914 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646552242913","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:56412","multi.service_calls":0,"queuetimems":28224,"class":"HRegionServer","multi.mutations":2} 2022-03-06 07:47:24,698 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646552544697","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:46888","multi.service_calls":0,"queuetimems":20242,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:47:25,240 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646552545239","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:56426","multi.service_calls":0,"queuetimems":1555,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:47:25,240 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646552545239","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:56426","multi.service_calls":0,"queuetimems":8611,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:47:25,786 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646552545786","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:56426","multi.service_calls":0,"queuetimems":1458,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:47:34,131 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646552554130","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:56426","multi.service_calls":0,"queuetimems":9432,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:47:34,721 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646552554721","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:56426","multi.service_calls":0,"queuetimems":10023,"class":"HRegionServer","multi.mutations":2} 2022-03-06 07:47:34,752 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646552554752","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:39712","multi.service_calls":0,"queuetimems":10024,"class":"HRegionServer","multi.mutations":2} 2022-03-06 07:47:34,774 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646552554773","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:56426","multi.service_calls":0,"queuetimems":9831,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:47:56,697 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646552576696","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:39718","multi.service_calls":0,"queuetimems":31133,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:47:56,932 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646552576932","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:46888","multi.service_calls":0,"queuetimems":27509,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:48:35,354 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2485ms No GCs detected 2022-03-06 07:48:46,017 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1187ms No GCs detected 2022-03-06 07:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=59453, hits=51354, hitRatio=86.38%, , cachingAccesses=51473, cachingHits=51045, cachingHitsRatio=99.17%, evictions=10156, evicted=0, evictedPerRun=0.0 2022-03-06 07:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 07:51:54,692 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646552814691","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:39772","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:51:59,727 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1290ms No GCs detected 2022-03-06 07:52:03,271 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646552823270","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:56444","multi.service_calls":0,"queuetimems":8579,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:52:04,456 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646552824455","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.30.41.118:39768","multi.service_calls":0,"queuetimems":9734,"class":"HRegionServer","multi.mutations":2} 2022-03-06 07:52:24,697 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646551944695","responsesize":"16534","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 10 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.118:39572","multi.service_calls":0,"queuetimems":21426,"class":"HRegionServer","multi.mutations":10} 2022-03-06 07:52:25,241 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646552845240","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:39772","multi.service_calls":0,"queuetimems":21969,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:52:25,641 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646552245639","responsesize":"18183","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 11 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900002,"client":"172.31.0.219:46872","multi.service_calls":0,"queuetimems":10,"class":"HRegionServer","multi.mutations":11} 2022-03-06 07:52:25,787 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646552845786","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:46904","multi.service_calls":0,"queuetimems":31095,"class":"HRegionServer","multi.mutations":2} 2022-03-06 07:52:34,132 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646552854131","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:39772","multi.service_calls":0,"queuetimems":30860,"class":"HRegionServer","multi.mutations":3} 2022-03-06 07:52:34,722 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646552854721","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:46904","multi.service_calls":0,"queuetimems":31450,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:52:34,753 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646552854752","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:46904","multi.service_calls":0,"queuetimems":31481,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:52:34,774 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646552854774","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:56444","multi.service_calls":0,"queuetimems":31503,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:52:56,933 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646552876932","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300001,"client":"172.30.41.118:39782","multi.service_calls":0,"queuetimems":215,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:54:33,528 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1287ms No GCs detected 2022-03-06 07:54:37,560 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 07:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=60167, hits=52068, hitRatio=86.54%, , cachingAccesses=52187, cachingHits=51759, cachingHitsRatio=99.18%, evictions=10186, evicted=0, evictedPerRun=0.0 2022-03-06 07:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 07:56:54,693 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646553114692","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:56460","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:57:03,272 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646553123272","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:39838","multi.service_calls":0,"queuetimems":8579,"class":"HRegionServer","multi.mutations":2} 2022-03-06 07:57:24,699 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646552844698","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:56444","multi.service_calls":0,"queuetimems":30007,"class":"HRegionServer","multi.mutations":5} 2022-03-06 07:57:25,641 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646553145641","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:46922","multi.service_calls":0,"queuetimems":22369,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:57:34,132 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646553154132","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300000,"client":"172.30.41.118:39838","multi.service_calls":0,"queuetimems":30860,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:57:34,723 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646553154723","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:46924","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:57:34,755 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646553154754","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:46924","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 07:57:34,775 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646553154774","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:46924","multi.service_calls":0,"queuetimems":20,"class":"HRegionServer","multi.mutations":2} 2022-03-06 07:57:56,698 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646552876697","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:56446","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-06 07:57:56,934 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646553176934","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:39848","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 08:00:33,406 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1158ms No GCs detected 2022-03-06 08:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=60947, hits=52848, hitRatio=86.71%, , cachingAccesses=52967, cachingHits=52539, cachingHitsRatio=99.19%, evictions=10216, evicted=0, evictedPerRun=0.0 2022-03-06 08:01:48,035 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 08:02:22,917 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646552842914","responsesize":"19835","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 12 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900003,"client":"172.30.41.118:39772","multi.service_calls":0,"queuetimems":28223,"class":"HRegionServer","multi.mutations":12} 2022-03-06 08:02:24,698 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646553144697","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 4 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:39838","multi.service_calls":0,"queuetimems":30004,"class":"HRegionServer","multi.mutations":4} 2022-03-06 08:02:24,699 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646553444699","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:56478","multi.service_calls":0,"queuetimems":21426,"class":"HRegionServer","multi.mutations":1} 2022-03-06 08:02:25,242 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646553145241","responsesize":"16528","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 10 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":600000,"client":"172.30.41.118:39838","multi.service_calls":0,"queuetimems":21969,"class":"HRegionServer","multi.mutations":10} 2022-03-06 08:02:25,788 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646553145787","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:46922","multi.service_calls":0,"queuetimems":31094,"class":"HRegionServer","multi.mutations":4} 2022-03-06 08:02:34,756 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646553454756","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:39924","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-06 08:02:34,775 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646553454775","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:56480","multi.service_calls":0,"queuetimems":19,"class":"HRegionServer","multi.mutations":1} 2022-03-06 08:02:56,699 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646553476699","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:46940","multi.service_calls":0,"queuetimems":21943,"class":"HRegionServer","multi.mutations":1} 2022-03-06 08:02:56,935 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646553476934","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:46940","multi.service_calls":0,"queuetimems":22178,"class":"HRegionServer","multi.mutations":1} 2022-03-06 08:03:49,500 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1287ms No GCs detected 2022-03-06 08:04:09,882 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1050ms No GCs detected 2022-03-06 08:05:00,191 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 5425ms No GCs detected 2022-03-06 08:05:35,312 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2791ms No GCs detected 2022-03-06 08:05:46,318 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 08:06:12,478 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1617ms No GCs detected 2022-03-06 08:06:16,545 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 08:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=61630, hits=53531, hitRatio=86.86%, , cachingAccesses=53650, cachingHits=53222, cachingHitsRatio=99.20%, evictions=10246, evicted=0, evictedPerRun=0.0 2022-03-06 08:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 08:07:03,275 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646553423273","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 5 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.155:56478","multi.service_calls":0,"queuetimems":8579,"class":"HRegionServer","multi.mutations":5} 2022-03-06 08:07:04,457 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646553124456","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 5 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:56460","multi.service_calls":0,"queuetimems":9763,"class":"HRegionServer","multi.mutations":5} 2022-03-06 08:07:22,919 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646553742918","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:56498","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 08:07:24,700 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646553744700","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:39998","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 08:07:25,242 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646552845240","responsesize":"11578","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 7 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200002,"client":"172.31.0.219:46904","multi.service_calls":0,"queuetimems":30549,"class":"HRegionServer","multi.mutations":7} 2022-03-06 08:07:25,642 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646553445641","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 5 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:39912","multi.service_calls":0,"queuetimems":30947,"class":"HRegionServer","multi.mutations":5} 2022-03-06 08:07:34,134 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646553454133","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:46940","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-06 08:07:34,724 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646553454724","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.31.0.219:46940","multi.service_calls":0,"queuetimems":591,"class":"HRegionServer","multi.mutations":3} 2022-03-06 08:07:34,757 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646553754756","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.30.41.118:39994","multi.service_calls":0,"queuetimems":11808,"class":"HRegionServer","multi.mutations":2} 2022-03-06 08:07:56,734 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646553776699","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300035,"client":"172.30.41.118:39998","multi.service_calls":0,"queuetimems":31457,"class":"HRegionServer","multi.mutations":1} 2022-03-06 08:07:56,935 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646553776935","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:39998","multi.service_calls":0,"queuetimems":31692,"class":"HRegionServer","multi.mutations":2} 2022-03-06 08:08:03,387 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1427ms No GCs detected 2022-03-06 08:09:18,986 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1644ms No GCs detected 2022-03-06 08:10:03,041 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1562ms No GCs detected 2022-03-06 08:10:06,990 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3448ms No GCs detected 2022-03-06 08:10:10,156 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1133ms No GCs detected 2022-03-06 08:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=62367, hits=54268, hitRatio=87.01%, , cachingAccesses=54387, cachingHits=53959, cachingHitsRatio=99.21%, evictions=10275, evicted=0, evictedPerRun=0.0 2022-03-06 08:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 08:11:54,696 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646553414694","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 5 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:39912","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":5} 2022-03-06 08:12:24,699 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646553744698","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 8 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:46960","multi.service_calls":0,"queuetimems":1779,"class":"HRegionServer","multi.mutations":8} 2022-03-06 08:12:25,243 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646554045242","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:56510","multi.service_calls":0,"queuetimems":2322,"class":"HRegionServer","multi.mutations":1} 2022-03-06 08:12:25,243 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646553745242","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:46960","multi.service_calls":0,"queuetimems":2323,"class":"HRegionServer","multi.mutations":3} 2022-03-06 08:12:25,643 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646554045642","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:46978","multi.service_calls":0,"queuetimems":22366,"class":"HRegionServer","multi.mutations":1} 2022-03-06 08:12:25,789 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646553745788","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:56498","multi.service_calls":0,"queuetimems":545,"class":"HRegionServer","multi.mutations":2} 2022-03-06 08:12:34,725 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646554054725","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:56510","multi.service_calls":0,"queuetimems":11805,"class":"HRegionServer","multi.mutations":3} 2022-03-06 08:12:41,056 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1888ms No GCs detected 2022-03-06 08:12:56,737 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646554076736","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:46980","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-06 08:12:56,936 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646554076935","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:46980","multi.service_calls":0,"queuetimems":200,"class":"HRegionServer","multi.mutations":1} 2022-03-06 08:13:09,505 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1919ms No GCs detected 2022-03-06 08:13:29,845 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1067ms No GCs detected 2022-03-06 08:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=63096, hits=54997, hitRatio=87.16%, , cachingAccesses=55116, cachingHits=54688, cachingHitsRatio=99.22%, evictions=10305, evicted=0, evictedPerRun=0.0 2022-03-06 08:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 08:17:03,277 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646554023276","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:56510","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 08:17:04,459 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646554024458","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:40062","multi.service_calls":0,"queuetimems":1182,"class":"HRegionServer","multi.mutations":3} 2022-03-06 08:17:22,921 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646554042919","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.118:40062","multi.service_calls":0,"queuetimems":19643,"class":"HRegionServer","multi.mutations":4} 2022-03-06 08:17:25,243 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646554345243","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:56530","multi.service_calls":0,"queuetimems":539,"class":"HRegionServer","multi.mutations":1} 2022-03-06 08:17:26,185 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646554345643","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300541,"client":"172.31.0.219:46998","multi.service_calls":0,"queuetimems":938,"class":"HRegionServer","multi.mutations":2} 2022-03-06 08:17:26,185 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646554345789","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300396,"client":"172.31.0.219:46998","multi.service_calls":0,"queuetimems":1084,"class":"HRegionServer","multi.mutations":1} 2022-03-06 08:17:34,728 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646554354727","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:40136","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-06 08:17:37,718 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646553754775","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 7 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":902943,"client":"172.30.41.118:39998","multi.service_calls":0,"queuetimems":10075,"class":"HRegionServer","multi.mutations":7} 2022-03-06 08:17:56,738 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646554376737","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:40136","multi.service_calls":0,"queuetimems":22010,"class":"HRegionServer","multi.mutations":1} 2022-03-06 08:17:56,937 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646554376936","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300001,"client":"172.30.41.118:40136","multi.service_calls":0,"queuetimems":22209,"class":"HRegionServer","multi.mutations":1} 2022-03-06 08:21:32,765 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1582ms No GCs detected 2022-03-06 08:21:40,396 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2260ms No GCs detected 2022-03-06 08:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=63983, hits=55884, hitRatio=87.34%, , cachingAccesses=56003, cachingHits=55575, cachingHitsRatio=99.24%, evictions=10335, evicted=0, evictedPerRun=0.0 2022-03-06 08:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 08:21:54,697 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646554314696","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:40126","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 08:22:03,279 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646554623278","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:40196","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 08:22:25,052 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646554642922","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":302130,"client":"172.30.41.118:40196","multi.service_calls":0,"queuetimems":19644,"class":"HRegionServer","multi.mutations":2} 2022-03-06 08:22:25,808 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646554645243","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300565,"client":"172.31.0.219:47024","multi.service_calls":0,"queuetimems":21934,"class":"HRegionServer","multi.mutations":2} 2022-03-06 08:22:25,872 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646554344704","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 4 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":601168,"client":"172.30.41.118:40136","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-06 08:22:26,471 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646554345243","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 4 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":601228,"client":"172.30.41.118:40136","multi.service_calls":0,"queuetimems":539,"class":"HRegionServer","multi.mutations":4} 2022-03-06 08:22:26,472 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1022ms No GCs detected 2022-03-06 08:22:26,598 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646554646185","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300413,"client":"172.30.41.155:56546","multi.service_calls":0,"queuetimems":22475,"class":"HRegionServer","multi.mutations":1} 2022-03-06 08:22:31,795 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1836ms No GCs detected 2022-03-06 08:22:34,136 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646554054135","responsesize":"18183","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 11 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.30.41.118:40062","multi.service_calls":0,"queuetimems":11215,"class":"HRegionServer","multi.mutations":11} 2022-03-06 08:22:34,730 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646554654728","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.155:56546","multi.service_calls":0,"queuetimems":31018,"class":"HRegionServer","multi.mutations":1} 2022-03-06 08:22:37,718 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646554657718","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:56546","multi.service_calls":0,"queuetimems":34008,"class":"HRegionServer","multi.mutations":1} 2022-03-06 08:22:37,823 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646554054757","responsesize":"19835","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 12 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":903066,"client":"172.31.0.219:46978","multi.service_calls":0,"queuetimems":31481,"class":"HRegionServer","multi.mutations":12} 2022-03-06 08:22:45,074 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2653ms No GCs detected 2022-03-06 08:22:56,738 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646554676738","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:56546","multi.service_calls":0,"queuetimems":53028,"class":"HRegionServer","multi.mutations":1} 2022-03-06 08:22:56,937 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646554676937","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:56546","multi.service_calls":0,"queuetimems":53227,"class":"HRegionServer","multi.mutations":1} 2022-03-06 08:23:03,215 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 08:26:44,457 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=64791, hits=56692, hitRatio=87.50%, , cachingAccesses=56811, cachingHits=56383, cachingHitsRatio=99.25%, evictions=10365, evicted=0, evictedPerRun=0.0 2022-03-06 08:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 08:27:03,420 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646554923280","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300140,"client":"172.30.41.118:40264","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 08:27:04,460 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646554624459","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.31.0.219:47020","multi.service_calls":0,"queuetimems":1181,"class":"HRegionServer","multi.mutations":2} 2022-03-06 08:27:16,526 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1123ms No GCs detected 2022-03-06 08:27:25,873 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646554044700","responsesize":"9926","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 6 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1201173,"client":"172.30.41.155:56510","multi.service_calls":0,"queuetimems":21424,"class":"HRegionServer","multi.mutations":6} 2022-03-06 08:27:26,753 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646554946752","responsesize":"6613","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:47060","multi.service_calls":0,"queuetimems":32053,"class":"HRegionServer","multi.mutations":4} 2022-03-06 08:27:37,823 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646554957823","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:40278","multi.service_calls":0,"queuetimems":3679,"class":"HRegionServer","multi.mutations":1} 2022-03-06 08:27:56,739 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646554976738","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:40278","multi.service_calls":0,"queuetimems":22594,"class":"HRegionServer","multi.mutations":1} 2022-03-06 08:27:56,938 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646554976937","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:40278","multi.service_calls":0,"queuetimems":22793,"class":"HRegionServer","multi.mutations":1} 2022-03-06 08:29:17,514 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3151ms No GCs detected 2022-03-06 08:29:37,169 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3073ms No GCs detected 2022-03-06 08:29:37,185 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 08:30:06,889 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3404ms No GCs detected 2022-03-06 08:30:22,152 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1145ms No GCs detected 2022-03-06 08:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=65527, hits=57428, hitRatio=87.64%, , cachingAccesses=57547, cachingHits=57119, cachingHitsRatio=99.26%, evictions=10395, evicted=0, evictedPerRun=0.0 2022-03-06 08:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-06 08:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 08:31:54,854 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646554914699","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600154,"client":"172.30.41.155:56568","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":4} 2022-03-06 08:32:04,460 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646555224460","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:40320","multi.service_calls":0,"queuetimems":1038,"class":"HRegionServer","multi.mutations":1} 2022-03-06 08:32:25,053 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646554945052","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":600001,"client":"172.30.41.118:40264","multi.service_calls":0,"queuetimems":30353,"class":"HRegionServer","multi.mutations":3} 2022-03-06 08:32:25,873 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646554945872","responsesize":"11572","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:47060","multi.service_calls":0,"queuetimems":31173,"class":"HRegionServer","multi.mutations":7} 2022-03-06 08:32:26,472 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646554946471","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:47060","multi.service_calls":0,"queuetimems":23191,"class":"HRegionServer","multi.mutations":3} 2022-03-06 08:32:34,768 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646554954138","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600630,"client":"172.30.41.155:56570","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":2} 2022-03-06 08:32:37,720 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646554957719","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 5 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:47062","multi.service_calls":0,"queuetimems":3575,"class":"HRegionServer","multi.mutations":5} 2022-03-06 08:32:56,938 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646555276938","responsesize":"9917","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:40320","multi.service_calls":0,"queuetimems":31064,"class":"HRegionServer","multi.mutations":6} 2022-03-06 08:34:20,296 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1622ms No GCs detected 2022-03-06 08:36:14,494 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1599ms No GCs detected 2022-03-06 08:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=66216, hits=58117, hitRatio=87.77%, , cachingAccesses=58236, cachingHits=57808, cachingHitsRatio=99.27%, evictions=10425, evicted=0, evictedPerRun=0.0 2022-03-06 08:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 08:37:03,423 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646555223422","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 8 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:47094","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":8} 2022-03-06 08:37:04,461 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646555524461","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:40392","multi.service_calls":0,"queuetimems":9606,"class":"HRegionServer","multi.mutations":2} 2022-03-06 08:37:25,809 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646554945808","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 7 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:40264","multi.service_calls":0,"queuetimems":31109,"class":"HRegionServer","multi.mutations":7} 2022-03-06 08:37:25,874 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646555545873","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:56602","multi.service_calls":0,"queuetimems":21412,"class":"HRegionServer","multi.mutations":1} 2022-03-06 08:37:25,874 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646555245874","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.31.0.219:47094","multi.service_calls":0,"queuetimems":22452,"class":"HRegionServer","multi.mutations":4} 2022-03-06 08:37:26,472 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646554646185","responsesize":"21490","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 13 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200287,"client":"172.31.0.219:47020","multi.service_calls":0,"queuetimems":22907,"class":"HRegionServer","multi.mutations":13} 2022-03-06 08:37:34,791 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646554954730","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 7 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900061,"client":"172.30.41.155:56570","multi.service_calls":0,"queuetimems":592,"class":"HRegionServer","multi.mutations":7} 2022-03-06 08:37:37,721 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646555557720","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:56604","multi.service_calls":0,"queuetimems":2952,"class":"HRegionServer","multi.mutations":1} 2022-03-06 08:37:56,740 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646555276739","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:56586","multi.service_calls":0,"queuetimems":50577,"class":"HRegionServer","multi.mutations":2} 2022-03-06 08:38:43,315 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1451ms No GCs detected 2022-03-06 08:39:19,254 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1015ms No GCs detected 2022-03-06 08:41:00,700 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1348ms No GCs detected 2022-03-06 08:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=66999, hits=58900, hitRatio=87.91%, , cachingAccesses=59019, cachingHits=58591, cachingHitsRatio=99.27%, evictions=10455, evicted=0, evictedPerRun=0.0 2022-03-06 08:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 08:41:54,921 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646555514855","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600065,"client":"172.30.41.155:56602","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":2} 2022-03-06 08:42:03,425 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646555823425","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:56620","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 08:42:04,462 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646555824461","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:56620","multi.service_calls":0,"queuetimems":1036,"class":"HRegionServer","multi.mutations":1} 2022-03-06 08:42:25,055 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646555545053","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.31.0.219:47112","multi.service_calls":0,"queuetimems":30198,"class":"HRegionServer","multi.mutations":2} 2022-03-06 08:42:25,811 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646555845810","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:40466","multi.service_calls":0,"queuetimems":22385,"class":"HRegionServer","multi.mutations":1} 2022-03-06 08:42:25,875 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646555845874","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:56620","multi.service_calls":0,"queuetimems":22449,"class":"HRegionServer","multi.mutations":1} 2022-03-06 08:42:26,473 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646555846472","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:40466","multi.service_calls":0,"queuetimems":23047,"class":"HRegionServer","multi.mutations":1} 2022-03-06 08:42:26,754 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646555246753","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:56586","multi.service_calls":0,"queuetimems":20591,"class":"HRegionServer","multi.mutations":6} 2022-03-06 08:42:34,769 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646555554768","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:40402","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-06 08:42:34,792 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646555854791","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:47130","multi.service_calls":0,"queuetimems":31366,"class":"HRegionServer","multi.mutations":1} 2022-03-06 08:42:37,721 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646555857721","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:56620","multi.service_calls":0,"queuetimems":34296,"class":"HRegionServer","multi.mutations":1} 2022-03-06 08:42:56,741 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646555876740","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:40466","multi.service_calls":0,"queuetimems":30930,"class":"HRegionServer","multi.mutations":1} 2022-03-06 08:42:56,939 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646555576939","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:40402","multi.service_calls":0,"queuetimems":22170,"class":"HRegionServer","multi.mutations":4} 2022-03-06 08:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=67712, hits=59613, hitRatio=88.04%, , cachingAccesses=59732, cachingHits=59304, cachingHitsRatio=99.28%, evictions=10485, evicted=0, evictedPerRun=0.0 2022-03-06 08:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 08:47:03,426 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646556123426","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:56640","multi.service_calls":0,"queuetimems":8457,"class":"HRegionServer","multi.mutations":2} 2022-03-06 08:47:04,462 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646556124462","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:40534","multi.service_calls":0,"queuetimems":9487,"class":"HRegionServer","multi.mutations":1} 2022-03-06 08:47:25,056 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646556145055","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:40534","multi.service_calls":0,"queuetimems":30081,"class":"HRegionServer","multi.mutations":1} 2022-03-06 08:47:25,812 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646556145811","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:40534","multi.service_calls":0,"queuetimems":30837,"class":"HRegionServer","multi.mutations":1} 2022-03-06 08:47:25,875 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646556145875","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:40534","multi.service_calls":0,"queuetimems":30901,"class":"HRegionServer","multi.mutations":1} 2022-03-06 08:47:25,875 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646555845875","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:56620","multi.service_calls":0,"queuetimems":22450,"class":"HRegionServer","multi.mutations":3} 2022-03-06 08:47:26,473 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646555546472","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 7 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:56602","multi.service_calls":0,"queuetimems":31617,"class":"HRegionServer","multi.mutations":7} 2022-03-06 08:47:26,473 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646556146473","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:40534","multi.service_calls":0,"queuetimems":31498,"class":"HRegionServer","multi.mutations":1} 2022-03-06 08:47:26,755 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646556146754","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300001,"client":"172.30.41.118:40534","multi.service_calls":0,"queuetimems":31779,"class":"HRegionServer","multi.mutations":1} 2022-03-06 08:47:34,771 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646556154770","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:40548","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 08:47:37,722 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646556157722","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:56642","multi.service_calls":0,"queuetimems":2952,"class":"HRegionServer","multi.mutations":2} 2022-03-06 08:47:37,826 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646555257823","responsesize":"16534","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 10 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200003,"client":"172.30.41.118:40320","multi.service_calls":0,"queuetimems":34401,"class":"HRegionServer","multi.mutations":10} 2022-03-06 08:47:52,661 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1975ms No GCs detected 2022-03-06 08:49:07,294 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4490ms No GCs detected 2022-03-06 08:49:36,091 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3043ms No GCs detected 2022-03-06 08:49:42,160 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 08:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=68515, hits=60416, hitRatio=88.18%, , cachingAccesses=60535, cachingHits=60107, cachingHitsRatio=99.29%, evictions=10515, evicted=0, evictedPerRun=0.0 2022-03-06 08:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 08:51:54,973 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646556114972","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:56640","multi.service_calls":0,"queuetimems":3,"class":"HRegionServer","multi.mutations":3} 2022-03-06 08:52:25,057 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646556445057","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:56658","multi.service_calls":0,"queuetimems":21630,"class":"HRegionServer","multi.mutations":1} 2022-03-06 08:52:25,812 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646556445812","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:47168","multi.service_calls":0,"queuetimems":22384,"class":"HRegionServer","multi.mutations":3} 2022-03-06 08:52:25,876 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646556445875","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:40610","multi.service_calls":0,"queuetimems":819,"class":"HRegionServer","multi.mutations":1} 2022-03-06 08:52:27,147 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646556446755","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300392,"client":"172.30.41.155:56658","multi.service_calls":0,"queuetimems":1698,"class":"HRegionServer","multi.mutations":1} 2022-03-06 08:52:34,793 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646556154792","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 5 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:40548","multi.service_calls":0,"queuetimems":22,"class":"HRegionServer","multi.mutations":5} 2022-03-06 08:52:37,723 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646556457723","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:40620","multi.service_calls":0,"queuetimems":11219,"class":"HRegionServer","multi.mutations":2} 2022-03-06 08:52:51,929 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1007ms No GCs detected 2022-03-06 08:52:57,758 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646556475943","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":301815,"client":"172.30.41.118:40622","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 08:52:57,758 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646556176940","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600818,"client":"172.30.41.155:56642","multi.service_calls":0,"queuetimems":19217,"class":"HRegionServer","multi.mutations":2} 2022-03-06 08:52:57,970 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 5541ms No GCs detected 2022-03-06 08:53:14,620 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1097ms No GCs detected 2022-03-06 08:53:18,221 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1065ms No GCs detected 2022-03-06 08:53:36,358 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4085ms No GCs detected 2022-03-06 08:53:48,440 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 08:54:20,266 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 08:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=69172, hits=61073, hitRatio=88.29%, , cachingAccesses=61192, cachingHits=60764, cachingHitsRatio=99.30%, evictions=10545, evicted=0, evictedPerRun=0.0 2022-03-06 08:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 08:57:04,464 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646556424463","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:40610","multi.service_calls":0,"queuetimems":1036,"class":"HRegionServer","multi.mutations":2} 2022-03-06 08:57:25,058 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646556745057","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:56678","multi.service_calls":0,"queuetimems":30083,"class":"HRegionServer","multi.mutations":2} 2022-03-06 08:57:25,813 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646556745812","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:47188","multi.service_calls":0,"queuetimems":30838,"class":"HRegionServer","multi.mutations":1} 2022-03-06 08:57:25,876 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646556445875","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 9 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:40610","multi.service_calls":0,"queuetimems":818,"class":"HRegionServer","multi.mutations":9} 2022-03-06 08:57:25,908 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646556745907","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:40692","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 08:57:28,104 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646556748103","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:56680","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 08:57:34,772 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646556454771","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:56658","multi.service_calls":0,"queuetimems":9714,"class":"HRegionServer","multi.mutations":4} 2022-03-06 08:57:37,725 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646556757724","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:40682","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 08:57:57,760 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646556777758","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.155:56680","multi.service_calls":0,"queuetimems":29655,"class":"HRegionServer","multi.mutations":1} 2022-03-06 08:59:06,385 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3569ms No GCs detected 2022-03-06 09:00:18,030 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2218ms No GCs detected 2022-03-06 09:00:28,698 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1158ms No GCs detected 2022-03-06 09:00:51,746 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 09:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=69959, hits=61860, hitRatio=88.42%, , cachingAccesses=61979, cachingHits=61551, cachingHitsRatio=99.31%, evictions=10574, evicted=0, evictedPerRun=0.0 2022-03-06 09:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 09:01:54,975 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646556714974","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:40682","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-06 09:02:03,430 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646556423427","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.118:40610","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":7} 2022-03-06 09:02:25,059 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646557045058","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:47202","multi.service_calls":0,"queuetimems":20593,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:02:25,814 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646557045813","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:47202","multi.service_calls":0,"queuetimems":21347,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:02:25,877 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646557045877","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:47202","multi.service_calls":0,"queuetimems":21411,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:02:25,908 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646557045908","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:47202","multi.service_calls":0,"queuetimems":21442,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:02:26,475 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646556446473","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 6 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.155:56658","multi.service_calls":0,"queuetimems":23045,"class":"HRegionServer","multi.mutations":6} 2022-03-06 09:02:28,105 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646557048104","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:56696","multi.service_calls":0,"queuetimems":3045,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:02:34,772 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646557054772","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300000,"client":"172.30.41.118:40756","multi.service_calls":0,"queuetimems":9713,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:02:57,760 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646556176741","responsesize":"14882","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 9 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1201019,"client":"172.31.0.219:47150","multi.service_calls":0,"queuetimems":21970,"class":"HRegionServer","multi.mutations":9} 2022-03-06 09:02:57,760 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646557077760","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:40756","multi.service_calls":0,"queuetimems":53295,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:02:57,760 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646556777758","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 8 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.31.0.219:47188","multi.service_calls":0,"queuetimems":62784,"class":"HRegionServer","multi.mutations":8} 2022-03-06 09:03:33,811 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1402ms No GCs detected 2022-03-06 09:03:35,448 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1136ms No GCs detected 2022-03-06 09:04:18,501 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1403ms No GCs detected 2022-03-06 09:06:40,015 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2440ms No GCs detected 2022-03-06 09:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=70684, hits=62585, hitRatio=88.54%, , cachingAccesses=62704, cachingHits=62276, cachingHitsRatio=99.32%, evictions=10604, evicted=0, evictedPerRun=0.0 2022-03-06 09:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 09:06:54,977 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646557314976","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:40816","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:07:04,466 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646557024465","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:56696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 09:07:25,814 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646557345814","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:56718","multi.service_calls":0,"queuetimems":30838,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:07:25,878 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646557345877","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:40816","multi.service_calls":0,"queuetimems":22447,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:07:25,909 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646557345908","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:56718","multi.service_calls":0,"queuetimems":22478,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:07:26,475 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646557346475","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:47222","multi.service_calls":0,"queuetimems":31498,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:07:26,475 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646556446474","responsesize":"21490","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 13 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200001,"client":"172.31.0.219:47168","multi.service_calls":0,"queuetimems":1416,"class":"HRegionServer","multi.mutations":13} 2022-03-06 09:07:28,105 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646557348105","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:56718","multi.service_calls":0,"queuetimems":24674,"class":"HRegionServer","multi.mutations":3} 2022-03-06 09:07:34,773 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646557354772","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.30.41.118:40824","multi.service_calls":0,"queuetimems":8864,"class":"HRegionServer","multi.mutations":2} 2022-03-06 09:07:37,725 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646556757723","responsesize":"18183","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 11 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900002,"client":"172.30.41.118:40682","multi.service_calls":0,"queuetimems":42749,"class":"HRegionServer","multi.mutations":11} 2022-03-06 09:07:57,761 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646557377761","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:40826","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:07:57,761 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646557377761","responsesize":"9917","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:40826","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":6} 2022-03-06 09:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=71506, hits=63407, hitRatio=88.67%, , cachingAccesses=63526, cachingHits=63098, cachingHitsRatio=99.33%, evictions=10634, evicted=0, evictedPerRun=0.0 2022-03-06 09:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 09:12:04,467 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646557624466","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:56730","multi.service_calls":0,"queuetimems":9488,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:12:25,815 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646557645814","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300001,"client":"172.30.41.118:40890","multi.service_calls":0,"queuetimems":30836,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:12:25,878 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646557645878","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:47240","multi.service_calls":0,"queuetimems":30900,"class":"HRegionServer","multi.mutations":2} 2022-03-06 09:12:25,909 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646557645909","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:47240","multi.service_calls":0,"queuetimems":30931,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:12:26,476 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646557646475","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:56730","multi.service_calls":0,"queuetimems":31497,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:12:26,476 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646557646475","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:56730","multi.service_calls":0,"queuetimems":31497,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:12:57,762 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646557377761","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:47224","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-06 09:12:57,763 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646557677762","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:40902","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-06 09:12:57,763 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646557677762","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:56736","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-06 09:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=72243, hits=64144, hitRatio=88.79%, , cachingAccesses=64263, cachingHits=63835, cachingHitsRatio=99.33%, evictions=10664, evicted=0, evictedPerRun=0.0 2022-03-06 09:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 09:16:54,980 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646557614978","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 6 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.155:56730","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":6} 2022-03-06 09:17:03,432 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646557323430","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:56718","multi.service_calls":0,"queuetimems":8454,"class":"HRegionServer","multi.mutations":7} 2022-03-06 09:17:04,468 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646557924468","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:56762","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:17:25,062 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646557345059","responsesize":"21487","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 13 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900002,"client":"172.30.41.118:40816","multi.service_calls":0,"queuetimems":30083,"class":"HRegionServer","multi.mutations":13} 2022-03-06 09:17:25,879 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646557945879","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:40954","multi.service_calls":0,"queuetimems":21411,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:17:26,476 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646557946476","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:47258","multi.service_calls":0,"queuetimems":22008,"class":"HRegionServer","multi.mutations":3} 2022-03-06 09:17:28,106 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646557648106","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:40890","multi.service_calls":0,"queuetimems":23639,"class":"HRegionServer","multi.mutations":4} 2022-03-06 09:17:34,774 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646557654773","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:56730","multi.service_calls":0,"queuetimems":30306,"class":"HRegionServer","multi.mutations":2} 2022-03-06 09:17:37,726 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646557657725","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 5 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:40890","multi.service_calls":0,"queuetimems":42747,"class":"HRegionServer","multi.mutations":5} 2022-03-06 09:17:37,726 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646557057725","responsesize":"24794","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 15 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200001,"client":"172.30.41.118:40756","multi.service_calls":0,"queuetimems":12666,"class":"HRegionServer","multi.mutations":15} 2022-03-06 09:17:57,763 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646557977762","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:56762","multi.service_calls":0,"queuetimems":31946,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:17:57,763 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646557977763","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:40954","multi.service_calls":0,"queuetimems":31947,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:17:57,763 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646557977763","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:47258","multi.service_calls":0,"queuetimems":31947,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=73114, hits=65015, hitRatio=88.92%, , cachingAccesses=65134, cachingHits=64706, cachingHitsRatio=99.34%, evictions=10694, evicted=0, evictedPerRun=0.0 2022-03-06 09:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 09:21:54,982 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646558214982","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:56800","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:22:03,433 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646558223432","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.30.41.118:40956","multi.service_calls":0,"queuetimems":10919,"class":"HRegionServer","multi.mutations":2} 2022-03-06 09:22:04,469 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646558224469","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:41008","multi.service_calls":0,"queuetimems":9487,"class":"HRegionServer","multi.mutations":2} 2022-03-06 09:22:25,062 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646558245062","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:56800","multi.service_calls":0,"queuetimems":30080,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:22:25,816 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646557945815","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 5 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:56762","multi.service_calls":0,"queuetimems":21347,"class":"HRegionServer","multi.mutations":5} 2022-03-06 09:22:25,880 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646558245879","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:56800","multi.service_calls":0,"queuetimems":29562,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:22:25,912 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646557945910","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 5 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.155:56762","multi.service_calls":0,"queuetimems":94,"class":"HRegionServer","multi.mutations":5} 2022-03-06 09:22:26,477 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646558246477","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:41008","multi.service_calls":0,"queuetimems":30995,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:22:28,107 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646558248107","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:47276","multi.service_calls":0,"queuetimems":27729,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:22:34,774 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646558254774","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:47276","multi.service_calls":0,"queuetimems":34356,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:22:37,727 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646558257726","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:47276","multi.service_calls":0,"queuetimems":39522,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:22:37,727 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646558257726","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:47276","multi.service_calls":0,"queuetimems":37109,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:22:57,763 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646558277763","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:47276","multi.service_calls":0,"queuetimems":57159,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:22:57,764 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646558277763","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:47276","multi.service_calls":0,"queuetimems":56994,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:22:57,764 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646558277763","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:47276","multi.service_calls":0,"queuetimems":57100,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=73840, hits=65741, hitRatio=89.03%, , cachingAccesses=65860, cachingHits=65432, cachingHitsRatio=99.35%, evictions=10724, evicted=0, evictedPerRun=0.0 2022-03-06 09:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 09:26:57,553 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646558514983","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":302569,"client":"172.30.41.118:41078","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:26:59,065 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3693ms No GCs detected 2022-03-06 09:27:03,434 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646558523433","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:47296","multi.service_calls":0,"queuetimems":8450,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:27:04,470 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646558524469","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.30.41.118:41074","multi.service_calls":0,"queuetimems":9456,"class":"HRegionServer","multi.mutations":2} 2022-03-06 09:27:25,063 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646558545062","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:56838","multi.service_calls":0,"queuetimems":30079,"class":"HRegionServer","multi.mutations":2} 2022-03-06 09:27:25,912 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646558545912","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:41078","multi.service_calls":0,"queuetimems":22478,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:27:26,477 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646558546477","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:56838","multi.service_calls":0,"queuetimems":23043,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:27:28,108 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646558548107","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:47296","multi.service_calls":0,"queuetimems":24673,"class":"HRegionServer","multi.mutations":2} 2022-03-06 09:27:32,087 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 09:27:37,727 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646558557727","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:56840","multi.service_calls":0,"queuetimems":2952,"class":"HRegionServer","multi.mutations":2} 2022-03-06 09:27:57,764 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646558577764","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300000,"client":"172.30.41.118:41088","multi.service_calls":0,"queuetimems":22969,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:27:57,764 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646558577764","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:41086","multi.service_calls":0,"queuetimems":20037,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=74578, hits=66479, hitRatio=89.14%, , cachingAccesses=66598, cachingHits=66170, cachingHitsRatio=99.36%, evictions=10754, evicted=0, evictedPerRun=0.0 2022-03-06 09:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-06 09:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 09:32:03,435 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646558823435","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:41140","multi.service_calls":0,"queuetimems":1596,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:32:04,471 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646558824470","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300001,"client":"172.30.41.118:41140","multi.service_calls":0,"queuetimems":2631,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:32:25,064 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646558845063","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:47318","multi.service_calls":0,"queuetimems":21627,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:32:25,817 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646558545816","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 5 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:41078","multi.service_calls":0,"queuetimems":30833,"class":"HRegionServer","multi.mutations":5} 2022-03-06 09:32:25,913 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646558845913","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:41140","multi.service_calls":0,"queuetimems":22475,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:32:26,478 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646557946476","responsesize":"26446","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 16 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Dev ","processingtimems":1200002,"client":"172.30.41.118:40954","multi.service_calls":0,"queuetimems":22008,"class":"HRegionServer","multi.mutations":16} 2022-03-06 09:32:26,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646558846478","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300000,"client":"172.30.41.118:41144","multi.service_calls":0,"queuetimems":22847,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:32:34,776 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646558554775","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:41086","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 09:32:37,728 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646558557727","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 8 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:41086","multi.service_calls":0,"queuetimems":2952,"class":"HRegionServer","multi.mutations":8} 2022-03-06 09:32:37,728 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646558857728","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:41140","multi.service_calls":0,"queuetimems":18305,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:32:57,765 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646558577763","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 9 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:47298","multi.service_calls":0,"queuetimems":22987,"class":"HRegionServer","multi.mutations":9} 2022-03-06 09:32:57,765 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646558877764","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:41140","multi.service_calls":0,"queuetimems":33275,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:32:57,766 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646558877766","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:56856","multi.service_calls":0,"queuetimems":31331,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:33:21,671 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 5456ms No GCs detected 2022-03-06 09:33:27,946 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1425ms No GCs detected 2022-03-06 09:33:32,952 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 09:34:02,819 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2306ms No GCs detected 2022-03-06 09:34:06,394 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3074ms No GCs detected 2022-03-06 09:35:27,205 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1073ms No GCs detected 2022-03-06 09:35:38,733 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 09:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=75338, hits=67239, hitRatio=89.25%, , cachingAccesses=67358, cachingHits=66930, cachingHitsRatio=99.36%, evictions=10784, evicted=0, evictedPerRun=0.0 2022-03-06 09:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 09:37:01,875 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646558819732","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 9 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":602143,"client":"172.31.0.219:47318","multi.service_calls":0,"queuetimems":657,"class":"HRegionServer","multi.mutations":9} 2022-03-06 09:37:04,471 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646559124471","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:47338","multi.service_calls":0,"queuetimems":1034,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:37:25,818 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646559145817","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.30.41.118:41216","multi.service_calls":0,"queuetimems":22351,"class":"HRegionServer","multi.mutations":2} 2022-03-06 09:37:25,881 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646558545880","responsesize":"21487","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 13 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.31.0.219:47296","multi.service_calls":0,"queuetimems":30896,"class":"HRegionServer","multi.mutations":13} 2022-03-06 09:37:25,914 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646559145913","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:47338","multi.service_calls":0,"queuetimems":848,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:37:26,479 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646559146478","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:41220","multi.service_calls":0,"queuetimems":1414,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:37:28,109 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646558848108","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:41140","multi.service_calls":0,"queuetimems":24670,"class":"HRegionServer","multi.mutations":3} 2022-03-06 09:37:37,730 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646559157728","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.118:41220","multi.service_calls":0,"queuetimems":12664,"class":"HRegionServer","multi.mutations":3} 2022-03-06 09:37:57,766 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646559177766","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:47340","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:37:57,766 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646559177766","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:56880","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:37:57,786 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646559177785","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300001,"client":"172.30.41.118:41224","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=76083, hits=67984, hitRatio=89.36%, , cachingAccesses=68103, cachingHits=67675, cachingHitsRatio=99.37%, evictions=10814, evicted=0, evictedPerRun=0.0 2022-03-06 09:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 09:42:01,877 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646559421876","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:41276","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:42:04,472 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646559424472","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:41276","multi.service_calls":0,"queuetimems":2596,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:42:25,065 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646559145064","responsesize":"11572","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 7 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:41220","multi.service_calls":0,"queuetimems":21627,"class":"HRegionServer","multi.mutations":7} 2022-03-06 09:42:25,819 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646559445818","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:41276","multi.service_calls":0,"queuetimems":23942,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:42:28,110 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646559448109","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:47356","multi.service_calls":0,"queuetimems":26232,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:42:34,777 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646559154776","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:56878","multi.service_calls":0,"queuetimems":8862,"class":"HRegionServer","multi.mutations":3} 2022-03-06 09:42:37,730 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646559157728","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 6 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.155:56878","multi.service_calls":0,"queuetimems":11249,"class":"HRegionServer","multi.mutations":6} 2022-03-06 09:42:57,767 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646559477766","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.31.0.219:47096","multi.service_calls":0,"queuetimems":5020,"class":"HRegionServer","multi.mutations":2} 2022-03-06 09:43:13,481 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3398ms No GCs detected 2022-03-06 09:43:36,876 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2148ms No GCs detected 2022-03-06 09:43:43,798 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 09:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=76794, hits=68695, hitRatio=89.45%, , cachingAccesses=68814, cachingHits=68386, cachingHitsRatio=99.38%, evictions=10844, evicted=0, evictedPerRun=0.0 2022-03-06 09:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 09:47:01,879 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646559721878","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:56916","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:47:04,474 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646559724473","responsesize":"9917","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 6 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:47388","multi.service_calls":0,"queuetimems":2595,"class":"HRegionServer","multi.mutations":6} 2022-03-06 09:47:25,915 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646559445914","responsesize":"11572","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 7 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:41276","multi.service_calls":0,"queuetimems":24038,"class":"HRegionServer","multi.mutations":7} 2022-03-06 09:47:26,480 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646559146478","responsesize":"16531","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 10 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.31.0.219:47338","multi.service_calls":0,"queuetimems":23041,"class":"HRegionServer","multi.mutations":10} 2022-03-06 09:47:26,480 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646559446479","responsesize":"11572","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 7 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:56898","multi.service_calls":0,"queuetimems":24603,"class":"HRegionServer","multi.mutations":7} 2022-03-06 09:47:57,787 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646559477786","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:56900","multi.service_calls":0,"queuetimems":19,"class":"HRegionServer","multi.mutations":2} 2022-03-06 09:49:04,896 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1160ms No GCs detected 2022-03-06 09:49:12,440 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 6277ms No GCs detected 2022-03-06 09:49:15,526 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 09:49:25,079 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1039ms No GCs detected 2022-03-06 09:49:31,410 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 5831ms No GCs detected 2022-03-06 09:49:53,994 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 5487ms No GCs detected 2022-03-06 09:50:03,689 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1148ms No GCs detected 2022-03-06 09:50:13,511 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1179ms No GCs detected 2022-03-06 09:50:17,048 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1095ms No GCs detected 2022-03-06 09:50:33,073 INFO [regionserver/cvp328:16201.Chore.3] hbase.ScheduledChore: Chore: MemstoreFlusherChore missed its start time 2022-03-06 09:50:33,073 INFO [regionserver/cvp328:16201.Chore.1] hbase.ScheduledChore: Chore: CompactionChecker missed its start time 2022-03-06 09:50:35,288 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 09:50:35,365 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 1 failover attempts. Trying to failover after sleeping for 464ms. 2022-03-06 09:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=77518, hits=69419, hitRatio=89.55%, , cachingAccesses=69538, cachingHits=69110, cachingHitsRatio=99.38%, evictions=10872, evicted=0, evictedPerRun=0.0 2022-03-06 09:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 09:52:01,881 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646560021880","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:41388","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-06 09:52:03,438 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646559123436","responsesize":"13230","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 8 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200002,"client":"172.30.41.118:41220","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":8} 2022-03-06 09:52:04,475 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646560024474","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:47428","multi.service_calls":0,"queuetimems":2594,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:52:25,068 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646559745065","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 6 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600003,"client":"172.30.41.118:41330","multi.service_calls":0,"queuetimems":23187,"class":"HRegionServer","multi.mutations":6} 2022-03-06 09:52:26,480 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646560046480","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:47428","multi.service_calls":0,"queuetimems":24600,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:52:27,653 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646559747652","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 5 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:56916","multi.service_calls":0,"queuetimems":25774,"class":"HRegionServer","multi.mutations":5} 2022-03-06 09:52:28,111 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646559748110","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 4 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:41330","multi.service_calls":0,"queuetimems":26232,"class":"HRegionServer","multi.mutations":4} 2022-03-06 09:52:34,779 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646559754778","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:41338","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-06 09:52:37,731 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646559757730","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 6 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:56918","multi.service_calls":0,"queuetimems":2952,"class":"HRegionServer","multi.mutations":6} 2022-03-06 09:52:37,733 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646559457730","responsesize":"21487","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 13 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900003,"client":"172.31.0.219:47356","multi.service_calls":0,"queuetimems":35853,"class":"HRegionServer","multi.mutations":13} 2022-03-06 09:52:57,768 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646559477767","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 5 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:56900","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":5} 2022-03-06 09:52:57,768 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646559777768","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 4 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.31.0.219:47394","multi.service_calls":0,"queuetimems":22990,"class":"HRegionServer","multi.mutations":4} 2022-03-06 09:52:57,788 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646560077788","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:56936","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:56:21,293 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 6035ms No GCs detected 2022-03-06 09:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=78206, hits=70107, hitRatio=89.64%, , cachingAccesses=70226, cachingHits=69798, cachingHitsRatio=99.39%, evictions=10902, evicted=0, evictedPerRun=0.0 2022-03-06 09:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 09:57:03,439 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646560323439","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:47466","multi.service_calls":0,"queuetimems":1557,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:57:04,476 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646560324475","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:56950","multi.service_calls":0,"queuetimems":2593,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:57:25,068 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646560345068","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:56950","multi.service_calls":0,"queuetimems":21629,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:57:25,916 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646560045915","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:56934","multi.service_calls":0,"queuetimems":24035,"class":"HRegionServer","multi.mutations":3} 2022-03-06 09:57:26,481 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646560046480","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 4 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:47428","multi.service_calls":0,"queuetimems":24600,"class":"HRegionServer","multi.mutations":4} 2022-03-06 09:57:27,653 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646560347653","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:47466","multi.service_calls":0,"queuetimems":24214,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:57:31,953 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646560351952","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.31.0.219:47468","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 09:57:37,734 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646560357733","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300001,"client":"172.30.41.118:41440","multi.service_calls":0,"queuetimems":12633,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:57:55,241 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646560375240","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300001,"client":"172.30.41.118:41450","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:57:57,769 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646560377769","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:56952","multi.service_calls":0,"queuetimems":22989,"class":"HRegionServer","multi.mutations":1} 2022-03-06 09:57:57,789 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646560377788","responsesize":"13221","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 8 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:47470","multi.service_calls":0,"queuetimems":23008,"class":"HRegionServer","multi.mutations":8} 2022-03-06 10:01:42,202 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1395ms No GCs detected 2022-03-06 10:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=78882, hits=70783, hitRatio=89.73%, , cachingAccesses=70902, cachingHits=70474, cachingHitsRatio=99.40%, evictions=10932, evicted=0, evictedPerRun=0.0 2022-03-06 10:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 10:02:03,441 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646560623440","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:41496","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:02:04,476 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646560624476","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:41496","multi.service_calls":0,"queuetimems":1036,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:02:25,070 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646560645069","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:41496","multi.service_calls":0,"queuetimems":21628,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:02:25,884 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646559445881","responsesize":"14885","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 9 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1500003,"client":"172.30.41.118:41276","multi.service_calls":0,"queuetimems":24005,"class":"HRegionServer","multi.mutations":9} 2022-03-06 10:02:25,917 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646560645916","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300000,"client":"172.30.41.118:41496","multi.service_calls":0,"queuetimems":22475,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:02:26,481 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646560346480","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 6 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:56950","multi.service_calls":0,"queuetimems":24598,"class":"HRegionServer","multi.mutations":6} 2022-03-06 10:02:26,481 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646560646481","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:41496","multi.service_calls":0,"queuetimems":23040,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:02:27,654 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646560647653","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:41496","multi.service_calls":0,"queuetimems":24212,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:02:31,953 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646560651953","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:41496","multi.service_calls":0,"queuetimems":28512,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:02:34,781 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646560354780","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 6 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:41448","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":6} 2022-03-06 10:02:37,736 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646560657734","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.118:41496","multi.service_calls":0,"queuetimems":34293,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:02:57,770 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646560677770","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:56974","multi.service_calls":0,"queuetimems":2528,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:02:57,790 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646560677789","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:56974","multi.service_calls":0,"queuetimems":2547,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:03:40,610 WARN [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 12537ms No GCs detected 2022-03-06 10:03:40,613 INFO [regionserver/cvp328:16201.Chore.3] hbase.ScheduledChore: Chore: MemstoreFlusherChore missed its start time 2022-03-06 10:03:40,613 INFO [regionserver/cvp328:16201.Chore.1] hbase.ScheduledChore: Chore: CompactionChecker missed its start time 2022-03-06 10:04:10,832 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 10:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=79631, hits=71532, hitRatio=89.83%, , cachingAccesses=71651, cachingHits=71223, cachingHitsRatio=99.40%, evictions=10961, evicted=0, evictedPerRun=0.0 2022-03-06 10:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 10:07:01,885 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646560321882","responsesize":"16531","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 10 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900003,"client":"172.30.41.118:41438","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":10} 2022-03-06 10:07:25,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646560945884","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:47532","multi.service_calls":0,"queuetimems":22442,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:07:27,686 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646560947686","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.31.0.219:47538","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 10:07:37,736 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646560957736","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:41542","multi.service_calls":0,"queuetimems":12665,"class":"HRegionServer","multi.mutations":2} 2022-03-06 10:07:55,243 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646560675241","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.118:41506","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 10:07:57,727 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646560977726","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.31.0.219:47540","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 10:07:57,740 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646560977740","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300000,"client":"172.30.41.118:41552","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:07:57,770 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646560377769","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:56952","multi.service_calls":0,"queuetimems":22989,"class":"HRegionServer","multi.mutations":7} 2022-03-06 10:07:57,771 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646560977770","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:47532","multi.service_calls":0,"queuetimems":32699,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:07:57,790 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646560977790","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:56990","multi.service_calls":0,"queuetimems":32719,"class":"HRegionServer","multi.mutations":2} 2022-03-06 10:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=80406, hits=72307, hitRatio=89.93%, , cachingAccesses=72426, cachingHits=71998, cachingHitsRatio=99.41%, evictions=10991, evicted=0, evictedPerRun=0.0 2022-03-06 10:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 10:12:01,887 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646561221886","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:41594","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:12:03,465 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646560923442","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600022,"client":"172.30.41.155:56990","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 10:12:25,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646561245885","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:57010","multi.service_calls":0,"queuetimems":23998,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:12:25,918 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646560945917","responsesize":"16528","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 10 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":600001,"client":"172.31.0.219:47532","multi.service_calls":0,"queuetimems":22474,"class":"HRegionServer","multi.mutations":10} 2022-03-06 10:12:26,482 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646560946482","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:56990","multi.service_calls":0,"queuetimems":1411,"class":"HRegionServer","multi.mutations":3} 2022-03-06 10:12:37,738 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646561257737","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:57010","multi.service_calls":0,"queuetimems":35851,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:12:55,243 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646561275243","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:57010","multi.service_calls":0,"queuetimems":53356,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:12:57,728 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646561277727","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:57010","multi.service_calls":0,"queuetimems":55840,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:12:57,742 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646561277741","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:41594","multi.service_calls":0,"queuetimems":20004,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:12:57,771 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646561277770","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:41594","multi.service_calls":0,"queuetimems":55884,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:12:57,771 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646561277771","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:41594","multi.service_calls":0,"queuetimems":55884,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:12:57,791 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646561277791","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:57010","multi.service_calls":0,"queuetimems":55904,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=81192, hits=73093, hitRatio=90.02%, , cachingAccesses=73212, cachingHits=72784, cachingHitsRatio=99.42%, evictions=11021, evicted=0, evictedPerRun=0.0 2022-03-06 10:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 10:17:03,466 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646561523465","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:57026","multi.service_calls":0,"queuetimems":1577,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:17:04,829 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646560924476","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 5 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900353,"client":"172.30.41.155:56990","multi.service_calls":0,"queuetimems":1034,"class":"HRegionServer","multi.mutations":5} 2022-03-06 10:17:25,073 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646560945070","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 4 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900003,"client":"172.30.41.118:41542","multi.service_calls":0,"queuetimems":21628,"class":"HRegionServer","multi.mutations":4} 2022-03-06 10:17:25,919 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646561545918","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.30.41.118:41658","multi.service_calls":0,"queuetimems":24000,"class":"HRegionServer","multi.mutations":2} 2022-03-06 10:17:27,687 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646561247686","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:41594","multi.service_calls":0,"queuetimems":25799,"class":"HRegionServer","multi.mutations":3} 2022-03-06 10:17:55,245 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646561575245","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:57028","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:17:57,773 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646561577771","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.31.0.219:47606","multi.service_calls":0,"queuetimems":2526,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:17:57,792 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646561577792","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:41666","multi.service_calls":0,"queuetimems":2518,"class":"HRegionServer","multi.mutations":2} 2022-03-06 10:21:20,838 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2367ms No GCs detected 2022-03-06 10:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=81922, hits=73823, hitRatio=90.11%, , cachingAccesses=73942, cachingHits=73514, cachingHitsRatio=99.42%, evictions=11051, evicted=0, evictedPerRun=0.0 2022-03-06 10:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 10:22:01,890 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646561521888","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.118:41654","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-06 10:22:25,074 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646561845073","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:57044","multi.service_calls":0,"queuetimems":21606,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:22:25,886 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646561545885","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 9 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:41654","multi.service_calls":0,"queuetimems":23997,"class":"HRegionServer","multi.mutations":9} 2022-03-06 10:22:25,919 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646561845919","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300000,"client":"172.30.41.118:41724","multi.service_calls":0,"queuetimems":845,"class":"HRegionServer","multi.mutations":3} 2022-03-06 10:22:26,483 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646560946481","responsesize":"13230","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 8 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200002,"client":"172.30.41.118:41542","multi.service_calls":0,"queuetimems":23039,"class":"HRegionServer","multi.mutations":8} 2022-03-06 10:22:26,483 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646561546483","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:57026","multi.service_calls":0,"queuetimems":24595,"class":"HRegionServer","multi.mutations":2} 2022-03-06 10:22:27,688 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646561847687","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:57044","multi.service_calls":0,"queuetimems":2613,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:22:37,739 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646561557738","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:41654","multi.service_calls":0,"queuetimems":34272,"class":"HRegionServer","multi.mutations":2} 2022-03-06 10:22:57,729 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646561577728","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:41670","multi.service_calls":0,"queuetimems":2483,"class":"HRegionServer","multi.mutations":4} 2022-03-06 10:22:57,743 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646561577742","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 5 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:41670","multi.service_calls":0,"queuetimems":2497,"class":"HRegionServer","multi.mutations":5} 2022-03-06 10:22:57,774 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646561577771","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 6 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600003,"client":"172.30.41.155:57028","multi.service_calls":0,"queuetimems":2526,"class":"HRegionServer","multi.mutations":6} 2022-03-06 10:26:44,914 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=82671, hits=74572, hitRatio=90.20%, , cachingAccesses=74691, cachingHits=74263, cachingHitsRatio=99.43%, evictions=11081, evicted=0, evictedPerRun=0.0 2022-03-06 10:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 10:27:01,893 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646562121892","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:41798","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:27:05,207 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646561823467","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":601739,"client":"172.30.41.118:41724","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-06 10:27:25,887 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646562145886","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:57056","multi.service_calls":0,"queuetimems":23993,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:27:25,920 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646562145920","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:57056","multi.service_calls":0,"queuetimems":24027,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:27:26,484 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646562146484","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:41798","multi.service_calls":0,"queuetimems":24593,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:27:26,484 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646562146484","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:41798","multi.service_calls":0,"queuetimems":24593,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:27:37,739 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646562157739","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:41798","multi.service_calls":0,"queuetimems":35848,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:27:57,729 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646562177729","responsesize":"6613","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:57056","multi.service_calls":0,"queuetimems":31245,"class":"HRegionServer","multi.mutations":4} 2022-03-06 10:27:57,774 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646561877773","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:57044","multi.service_calls":0,"queuetimems":54306,"class":"HRegionServer","multi.mutations":2} 2022-03-06 10:27:57,774 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646562177774","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:41798","multi.service_calls":0,"queuetimems":55883,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=83408, hits=75309, hitRatio=90.29%, , cachingAccesses=75428, cachingHits=75000, cachingHitsRatio=99.43%, evictions=11111, evicted=0, evictedPerRun=0.0 2022-03-06 10:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-06 10:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 10:32:01,894 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646562421894","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:41868","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:32:05,208 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646562425207","responsesize":"9917","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:41868","multi.service_calls":0,"queuetimems":3313,"class":"HRegionServer","multi.mutations":6} 2022-03-06 10:32:24,078 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1047ms No GCs detected 2022-03-06 10:32:28,134 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3555ms No GCs detected 2022-03-06 10:32:28,145 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646562445887","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":302257,"client":"172.31.0.219:47658","multi.service_calls":0,"queuetimems":23993,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:32:28,177 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646562145074","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":603102,"client":"172.30.41.155:57056","multi.service_calls":0,"queuetimems":23181,"class":"HRegionServer","multi.mutations":3} 2022-03-06 10:32:28,145 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646562445920","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":302225,"client":"172.30.41.118:41872","multi.service_calls":0,"queuetimems":23996,"class":"HRegionServer","multi.mutations":2} 2022-03-06 10:32:28,188 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646562446484","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":301704,"client":"172.30.41.155:57074","multi.service_calls":0,"queuetimems":23807,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:32:36,339 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2061ms No GCs detected 2022-03-06 10:32:37,741 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646562457740","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:41882","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:32:55,248 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646561875246","responsesize":"16531","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 10 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.30.41.118:41734","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":10} 2022-03-06 10:32:57,746 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646562177744","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.31.0.219:47642","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-06 10:32:57,794 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646561877792","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 8 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:57044","multi.service_calls":0,"queuetimems":32718,"class":"HRegionServer","multi.mutations":8} 2022-03-06 10:36:19,391 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1838ms No GCs detected 2022-03-06 10:36:21,371 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1480ms No GCs detected 2022-03-06 10:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=84202, hits=76103, hitRatio=90.38%, , cachingAccesses=76222, cachingHits=75794, cachingHitsRatio=99.44%, evictions=11141, evicted=0, evictedPerRun=0.0 2022-03-06 10:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 10:36:52,078 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 10:37:01,896 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646562721895","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:57096","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:37:05,213 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646561824829","responsesize":"19838","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 12 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200384,"client":"172.30.41.118:41724","multi.service_calls":0,"queuetimems":1362,"class":"HRegionServer","multi.mutations":12} 2022-03-06 10:37:28,107 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646562446484","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":601623,"client":"172.31.0.219:47658","multi.service_calls":0,"queuetimems":24590,"class":"HRegionServer","multi.mutations":3} 2022-03-06 10:37:55,249 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646562775248","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:57098","multi.service_calls":0,"queuetimems":17506,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:37:57,731 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646562477730","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:57076","multi.service_calls":0,"queuetimems":19990,"class":"HRegionServer","multi.mutations":3} 2022-03-06 10:37:57,747 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646562777746","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:47678","multi.service_calls":0,"queuetimems":20004,"class":"HRegionServer","multi.mutations":3} 2022-03-06 10:37:57,775 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646562477774","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:47660","multi.service_calls":0,"queuetimems":20033,"class":"HRegionServer","multi.mutations":2} 2022-03-06 10:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=84993, hits=76894, hitRatio=90.47%, , cachingAccesses=77013, cachingHits=76585, cachingHitsRatio=99.44%, evictions=11171, evicted=0, evictedPerRun=0.0 2022-03-06 10:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 10:42:01,898 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646563021897","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:42020","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:42:05,210 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646562725208","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 5 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.118:41948","multi.service_calls":0,"queuetimems":3313,"class":"HRegionServer","multi.mutations":5} 2022-03-06 10:42:28,109 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646562147688","responsesize":"13230","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 8 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200421,"client":"172.31.0.219:47640","multi.service_calls":0,"queuetimems":25795,"class":"HRegionServer","multi.mutations":8} 2022-03-06 10:42:28,187 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646562748186","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:57096","multi.service_calls":0,"queuetimems":26291,"class":"HRegionServer","multi.mutations":2} 2022-03-06 10:42:37,743 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646562757742","responsesize":"11572","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 7 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:41958","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":7} 2022-03-06 10:42:57,779 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646563077778","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:42026","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 10:42:57,795 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646562777794","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 8 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:47678","multi.service_calls":0,"queuetimems":20052,"class":"HRegionServer","multi.mutations":8} 2022-03-06 10:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=85763, hits=77664, hitRatio=90.56%, , cachingAccesses=77783, cachingHits=77355, cachingHitsRatio=99.45%, evictions=11201, evicted=0, evictedPerRun=0.0 2022-03-06 10:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 10:47:01,900 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646563321899","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:42086","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:47:05,225 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646563025214","responsesize":"16528","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 10 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":600010,"client":"172.30.41.118:42020","multi.service_calls":0,"queuetimems":3317,"class":"HRegionServer","multi.mutations":10} 2022-03-06 10:47:28,110 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646563048107","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 6 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600003,"client":"172.30.41.155:57114","multi.service_calls":0,"queuetimems":26209,"class":"HRegionServer","multi.mutations":6} 2022-03-06 10:47:28,110 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646563348110","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:42086","multi.service_calls":0,"queuetimems":26211,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:47:28,188 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646562748186","responsesize":"18183","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 11 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.31.0.219:47676","multi.service_calls":0,"queuetimems":26290,"class":"HRegionServer","multi.mutations":11} 2022-03-06 10:47:28,189 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646562748188","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 7 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:41948","multi.service_calls":0,"queuetimems":26292,"class":"HRegionServer","multi.mutations":7} 2022-03-06 10:47:57,777 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646562477774","responsesize":"24794","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 15 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200003,"client":"172.30.41.118:41882","multi.service_calls":0,"queuetimems":20034,"class":"HRegionServer","multi.mutations":15} 2022-03-06 10:47:57,777 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646563077775","responsesize":"16528","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 10 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":600002,"client":"172.31.0.219:47694","multi.service_calls":0,"queuetimems":55877,"class":"HRegionServer","multi.mutations":10} 2022-03-06 10:47:57,780 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646563377779","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:47718","multi.service_calls":0,"queuetimems":20034,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:47:57,795 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646563377795","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:57132","multi.service_calls":0,"queuetimems":20050,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:51:18,779 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3730ms No GCs detected 2022-03-06 10:51:25,757 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 10:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=86546, hits=78447, hitRatio=90.64%, , cachingAccesses=78566, cachingHits=78138, cachingHitsRatio=99.46%, evictions=11231, evicted=0, evictedPerRun=0.0 2022-03-06 10:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 10:52:01,901 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646563621901","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:57150","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:52:28,110 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646563648110","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:57150","multi.service_calls":0,"queuetimems":26209,"class":"HRegionServer","multi.mutations":3} 2022-03-06 10:52:28,188 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646562748186","responsesize":"6622","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200002,"client":"172.31.0.219:47676","multi.service_calls":0,"queuetimems":26290,"class":"HRegionServer","multi.mutations":4} 2022-03-06 10:52:28,188 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646563348187","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:57130","multi.service_calls":0,"queuetimems":26288,"class":"HRegionServer","multi.mutations":3} 2022-03-06 10:52:28,188 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646563648188","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.31.0.219:47758","multi.service_calls":0,"queuetimems":26257,"class":"HRegionServer","multi.mutations":2} 2022-03-06 10:52:28,189 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646563648189","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:47754","multi.service_calls":0,"queuetimems":26288,"class":"HRegionServer","multi.mutations":3} 2022-03-06 10:52:37,746 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646563357745","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:57132","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-06 10:52:55,250 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646563075249","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:47694","multi.service_calls":0,"queuetimems":53351,"class":"HRegionServer","multi.mutations":6} 2022-03-06 10:52:57,733 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646563077731","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 7 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.118:42020","multi.service_calls":0,"queuetimems":52510,"class":"HRegionServer","multi.mutations":7} 2022-03-06 10:52:57,796 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646563677795","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:47762","multi.service_calls":0,"queuetimems":16,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=87315, hits=79216, hitRatio=90.72%, , cachingAccesses=79335, cachingHits=78907, cachingHitsRatio=99.46%, evictions=11261, evicted=0, evictedPerRun=0.0 2022-03-06 10:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 10:57:05,212 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646563325210","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.155:57130","multi.service_calls":0,"queuetimems":3311,"class":"HRegionServer","multi.mutations":4} 2022-03-06 10:57:28,112 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646563648110","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 5 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.118:42142","multi.service_calls":0,"queuetimems":26209,"class":"HRegionServer","multi.mutations":5} 2022-03-06 10:57:28,112 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646563948111","responsesize":"6613","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:47798","multi.service_calls":0,"queuetimems":26208,"class":"HRegionServer","multi.mutations":4} 2022-03-06 10:57:28,189 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646563948188","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:47798","multi.service_calls":0,"queuetimems":76,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:57:28,189 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646563948188","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:47798","multi.service_calls":0,"queuetimems":76,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:57:28,190 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646563948189","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:47798","multi.service_calls":0,"queuetimems":77,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:57:37,746 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646563957746","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:47798","multi.service_calls":0,"queuetimems":9634,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:57:57,733 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646563977733","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:57172","multi.service_calls":0,"queuetimems":29543,"class":"HRegionServer","multi.mutations":1} 2022-03-06 10:57:57,780 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646563677779","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:57152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-06 10:57:57,780 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646563677780","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:42148","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":4} 2022-03-06 10:57:57,796 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646563977796","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:47798","multi.service_calls":0,"queuetimems":29684,"class":"HRegionServer","multi.mutations":1} 2022-03-06 11:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=88093, hits=79994, hitRatio=90.81%, , cachingAccesses=80113, cachingHits=79685, cachingHitsRatio=99.47%, evictions=11291, evicted=0, evictedPerRun=0.0 2022-03-06 11:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 11:02:01,904 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646563921903","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:47798","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":5} 2022-03-06 11:02:05,213 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646564225213","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:42270","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 11:02:28,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646564248113","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:57190","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 11:02:28,190 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646563948190","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:57172","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":3} 2022-03-06 11:02:28,190 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646564248190","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:42270","multi.service_calls":0,"queuetimems":77,"class":"HRegionServer","multi.mutations":1} 2022-03-06 11:02:57,781 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646563677778","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 8 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.118:42148","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":8} 2022-03-06 11:02:57,781 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646564277780","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:57190","multi.service_calls":0,"queuetimems":29667,"class":"HRegionServer","multi.mutations":1} 2022-03-06 11:02:57,781 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646564277780","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:57190","multi.service_calls":0,"queuetimems":29667,"class":"HRegionServer","multi.mutations":1} 2022-03-06 11:02:57,797 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646564277796","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:57190","multi.service_calls":0,"queuetimems":29683,"class":"HRegionServer","multi.mutations":1} 2022-03-06 11:06:33,932 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1313ms No GCs detected 2022-03-06 11:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=88753, hits=80654, hitRatio=90.87%, , cachingAccesses=80773, cachingHits=80345, cachingHitsRatio=99.47%, evictions=11321, evicted=0, evictedPerRun=0.0 2022-03-06 11:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 11:07:01,906 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646564521905","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:42344","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 11:07:05,214 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646564525214","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:57214","multi.service_calls":0,"queuetimems":3309,"class":"HRegionServer","multi.mutations":1} 2022-03-06 11:07:05,233 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646563625226","responsesize":"16534","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 10 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200007,"client":"172.30.41.118:42142","multi.service_calls":0,"queuetimems":3325,"class":"HRegionServer","multi.mutations":10} 2022-03-06 11:07:28,113 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646564248112","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:57190","multi.service_calls":0,"queuetimems":22899,"class":"HRegionServer","multi.mutations":2} 2022-03-06 11:07:28,114 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646564548113","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:47832","multi.service_calls":0,"queuetimems":26207,"class":"HRegionServer","multi.mutations":1} 2022-03-06 11:07:28,190 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646564248189","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 6 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:57190","multi.service_calls":0,"queuetimems":22976,"class":"HRegionServer","multi.mutations":6} 2022-03-06 11:07:28,190 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646564248189","responsesize":"16528","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 10 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Dev ","processingtimems":600001,"client":"172.30.41.118:42270","multi.service_calls":0,"queuetimems":22976,"class":"HRegionServer","multi.mutations":10} 2022-03-06 11:07:28,191 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646564548190","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.30.41.118:42342","multi.service_calls":0,"queuetimems":26255,"class":"HRegionServer","multi.mutations":2} 2022-03-06 11:07:55,252 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646563975251","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 5 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:57172","multi.service_calls":0,"queuetimems":27062,"class":"HRegionServer","multi.mutations":5} 2022-03-06 11:07:57,735 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646564277734","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.31.0.219:47814","multi.service_calls":0,"queuetimems":52521,"class":"HRegionServer","multi.mutations":3} 2022-03-06 11:07:57,782 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646564577782","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:42356","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 11:09:36,456 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1576ms No GCs detected 2022-03-06 11:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=89493, hits=81394, hitRatio=90.95%, , cachingAccesses=81513, cachingHits=81085, cachingHitsRatio=99.47%, evictions=11350, evicted=0, evictedPerRun=0.0 2022-03-06 11:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 11:12:05,237 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646564825233","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300004,"client":"172.31.0.219:47850","multi.service_calls":0,"queuetimems":3326,"class":"HRegionServer","multi.mutations":1} 2022-03-06 11:12:28,190 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646564848190","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:47850","multi.service_calls":0,"queuetimems":26283,"class":"HRegionServer","multi.mutations":1} 2022-03-06 11:12:33,654 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 11:12:37,748 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646564257747","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 7 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:42270","multi.service_calls":0,"queuetimems":9634,"class":"HRegionServer","multi.mutations":7} 2022-03-06 11:12:57,735 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646564877735","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:57232","multi.service_calls":0,"queuetimems":52520,"class":"HRegionServer","multi.mutations":1} 2022-03-06 11:12:57,783 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646564577782","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:57216","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 11:15:08,100 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 11:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=90133, hits=82034, hitRatio=91.01%, , cachingAccesses=82153, cachingHits=81725, cachingHitsRatio=99.48%, evictions=11380, evicted=0, evictedPerRun=0.0 2022-03-06 11:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 11:17:05,239 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646565125238","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:42484","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 11:17:28,116 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646564848114","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.155:57232","multi.service_calls":0,"queuetimems":22899,"class":"HRegionServer","multi.mutations":5} 2022-03-06 11:17:28,116 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646564848114","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.155:57232","multi.service_calls":0,"queuetimems":26207,"class":"HRegionServer","multi.mutations":3} 2022-03-06 11:17:28,191 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646564848191","responsesize":"11572","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.31.0.219:47850","multi.service_calls":0,"queuetimems":22975,"class":"HRegionServer","multi.mutations":7} 2022-03-06 11:17:28,191 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646565148191","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:42484","multi.service_calls":0,"queuetimems":22615,"class":"HRegionServer","multi.mutations":1} 2022-03-06 11:17:28,191 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646564548190","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:42344","multi.service_calls":0,"queuetimems":26285,"class":"HRegionServer","multi.mutations":7} 2022-03-06 11:17:37,749 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646565157748","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:42484","multi.service_calls":0,"queuetimems":32509,"class":"HRegionServer","multi.mutations":1} 2022-03-06 11:17:55,253 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646564875252","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:47850","multi.service_calls":0,"queuetimems":53345,"class":"HRegionServer","multi.mutations":4} 2022-03-06 11:17:57,736 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646565177735","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:57250","multi.service_calls":0,"queuetimems":38130,"class":"HRegionServer","multi.mutations":1} 2022-03-06 11:17:57,785 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646564877782","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 4 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600003,"client":"172.31.0.219:47850","multi.service_calls":0,"queuetimems":52566,"class":"HRegionServer","multi.mutations":4} 2022-03-06 11:17:57,785 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646565177783","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.31.0.219:47866","multi.service_calls":0,"queuetimems":51421,"class":"HRegionServer","multi.mutations":1} 2022-03-06 11:18:12,711 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1022ms No GCs detected 2022-03-06 11:18:12,734 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 11:18:22,322 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1962ms No GCs detected 2022-03-06 11:18:28,654 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1816ms No GCs detected 2022-03-06 11:18:43,654 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 11:20:18,749 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1388ms No GCs detected 2022-03-06 11:20:42,589 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1519ms No GCs detected 2022-03-06 11:21:03,449 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1615ms No GCs detected 2022-03-06 11:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=90864, hits=82765, hitRatio=91.09%, , cachingAccesses=82884, cachingHits=82456, cachingHitsRatio=99.48%, evictions=11410, evicted=0, evictedPerRun=0.0 2022-03-06 11:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 11:22:01,909 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646564821907","responsesize":"14879","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 9 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.118:42408","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":9} 2022-03-06 11:22:05,224 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646564825215","responsesize":"16531","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 10 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900009,"client":"172.30.41.118:42408","multi.service_calls":0,"queuetimems":3308,"class":"HRegionServer","multi.mutations":10} 2022-03-06 11:22:05,241 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646565425240","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:57268","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 11:22:13,930 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1443ms No GCs detected 2022-03-06 11:22:28,118 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646565448117","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:42546","multi.service_calls":0,"queuetimems":22877,"class":"HRegionServer","multi.mutations":1} 2022-03-06 11:22:28,191 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646564848190","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:57232","multi.service_calls":0,"queuetimems":26283,"class":"HRegionServer","multi.mutations":3} 2022-03-06 11:22:28,192 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646565448191","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.30.41.118:42544","multi.service_calls":0,"queuetimems":22921,"class":"HRegionServer","multi.mutations":2} 2022-03-06 11:22:57,737 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646565477736","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:47882","multi.service_calls":0,"queuetimems":29619,"class":"HRegionServer","multi.mutations":1} 2022-03-06 11:22:57,786 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646565477785","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:42546","multi.service_calls":0,"queuetimems":29668,"class":"HRegionServer","multi.mutations":1} 2022-03-06 11:22:57,786 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646565477785","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:57268","multi.service_calls":0,"queuetimems":29625,"class":"HRegionServer","multi.mutations":1} 2022-03-06 11:22:57,798 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646564577797","responsesize":"18186","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 11 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200001,"client":"172.31.0.219:47834","multi.service_calls":0,"queuetimems":14,"class":"HRegionServer","multi.mutations":11} 2022-03-06 11:23:34,588 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2404ms No GCs detected 2022-03-06 11:24:06,520 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2497ms No GCs detected 2022-03-06 11:26:25,703 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1155ms No GCs detected 2022-03-06 11:26:36,576 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 11:26:40,825 WARN [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 14621ms No GCs detected 2022-03-06 11:26:41,639 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 699ms. 2022-03-06 11:26:42,366 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: tNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 2 failover attempts. Trying to failover after sleeping for 1446ms. 2022-03-06 11:26:43,818 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: tNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 3 failover attempts. Trying to failover after sleeping for 1604ms. 2022-03-06 11:26:44,382 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=91606, hits=83507, hitRatio=91.16%, , cachingAccesses=83626, cachingHits=83198, cachingHitsRatio=99.49%, evictions=11440, evicted=0, evictedPerRun=0.0 2022-03-06 11:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 11:27:15,760 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 11:27:28,119 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646565448116","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 4 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.31.0.219:47882","multi.service_calls":0,"queuetimems":22876,"class":"HRegionServer","multi.mutations":4} 2022-03-06 11:27:28,119 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646565748118","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:42608","multi.service_calls":0,"queuetimems":26208,"class":"HRegionServer","multi.mutations":3} 2022-03-06 11:27:28,192 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646565448192","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 9 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:42546","multi.service_calls":0,"queuetimems":75,"class":"HRegionServer","multi.mutations":9} 2022-03-06 11:27:28,192 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646565748192","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:42608","multi.service_calls":0,"queuetimems":26282,"class":"HRegionServer","multi.mutations":1} 2022-03-06 11:27:37,750 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646565457749","responsesize":"16528","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 10 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":600001,"client":"172.31.0.219:47882","multi.service_calls":0,"queuetimems":32509,"class":"HRegionServer","multi.mutations":10} 2022-03-06 11:27:55,255 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646565475253","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.31.0.219:47882","multi.service_calls":0,"queuetimems":27136,"class":"HRegionServer","multi.mutations":2} 2022-03-06 11:27:57,786 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646564577782","responsesize":"16537","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 10 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1500004,"client":"172.30.41.155:57216","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":10} 2022-03-06 11:27:57,786 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646565777786","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:57292","multi.service_calls":0,"queuetimems":48,"class":"HRegionServer","multi.mutations":1} 2022-03-06 11:27:57,787 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646565777786","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.30.41.155:56836","multi.service_calls":0,"queuetimems":3882,"class":"HRegionServer","multi.mutations":2} 2022-03-06 11:27:57,799 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646565777798","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300001,"client":"172.30.41.118:42616","multi.service_calls":0,"queuetimems":11,"class":"HRegionServer","multi.mutations":1} 2022-03-06 11:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=92357, hits=84258, hitRatio=91.23%, , cachingAccesses=84377, cachingHits=83949, cachingHitsRatio=99.49%, evictions=11470, evicted=0, evictedPerRun=0.0 2022-03-06 11:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-06 11:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 11:31:58,660 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1052ms No GCs detected 2022-03-06 11:32:01,911 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646565721910","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:57290","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-06 11:32:05,226 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646565725225","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 6 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:47904","multi.service_calls":0,"queuetimems":3315,"class":"HRegionServer","multi.mutations":6} 2022-03-06 11:32:05,242 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646565725241","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.31.0.219:47904","multi.service_calls":0,"queuetimems":16,"class":"HRegionServer","multi.mutations":2} 2022-03-06 11:32:13,883 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3019ms No GCs detected 2022-03-06 11:32:18,897 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 11:32:19,451 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 1 failover attempts. Trying to failover after sleeping for 675ms. 2022-03-06 11:32:20,129 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: tNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 2 failover attempts. Trying to failover after sleeping for 1010ms. 2022-03-06 11:32:28,192 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646565748191","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:57290","multi.service_calls":0,"queuetimems":26281,"class":"HRegionServer","multi.mutations":2} 2022-03-06 11:32:28,193 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646565448191","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 8 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.118:42546","multi.service_calls":0,"queuetimems":22951,"class":"HRegionServer","multi.mutations":8} 2022-03-06 11:32:55,470 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646566075469","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300001,"client":"172.30.41.118:42678","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 11:32:57,738 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646565777738","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:42616","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-06 11:32:57,788 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646566077787","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.30.41.155:57326","multi.service_calls":0,"queuetimems":29637,"class":"HRegionServer","multi.mutations":2} 2022-03-06 11:32:59,060 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646566079060","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.155:57330","multi.service_calls":0,"queuetimems":2,"class":"HRegionServer","multi.mutations":2} 2022-03-06 11:33:20,388 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646566100387","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:47930","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 11:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=93084, hits=84985, hitRatio=91.30%, , cachingAccesses=85104, cachingHits=84676, cachingHitsRatio=99.50%, evictions=11500, evicted=0, evictedPerRun=0.0 2022-03-06 11:36:48,027 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 11:37:01,963 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646566321912","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300051,"client":"172.30.41.118:42714","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 11:37:05,227 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646566325226","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:42714","multi.service_calls":0,"queuetimems":3314,"class":"HRegionServer","multi.mutations":1} 2022-03-06 11:37:05,242 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646566325242","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:42714","multi.service_calls":0,"queuetimems":3330,"class":"HRegionServer","multi.mutations":1} 2022-03-06 11:37:28,193 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646566348192","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:42714","multi.service_calls":0,"queuetimems":26280,"class":"HRegionServer","multi.mutations":1} 2022-03-06 11:37:28,193 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646566348193","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:42714","multi.service_calls":0,"queuetimems":26280,"class":"HRegionServer","multi.mutations":1} 2022-03-06 11:37:28,193 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646566048192","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:47928","multi.service_calls":0,"queuetimems":72,"class":"HRegionServer","multi.mutations":4} 2022-03-06 11:37:55,471 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646566375471","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:42730","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 11:37:57,789 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646566377789","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:42730","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 11:38:20,389 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646566400388","responsesize":"6613","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:42730","multi.service_calls":0,"queuetimems":22599,"class":"HRegionServer","multi.mutations":4} 2022-03-06 11:39:09,223 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 6573ms No GCs detected 2022-03-06 11:40:01,474 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3423ms No GCs detected 2022-03-06 11:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=93746, hits=85647, hitRatio=91.36%, , cachingAccesses=85766, cachingHits=85338, cachingHitsRatio=99.50%, evictions=11529, evicted=0, evictedPerRun=0.0 2022-03-06 11:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 11:42:05,228 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646566625227","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:57390","multi.service_calls":0,"queuetimems":3262,"class":"HRegionServer","multi.mutations":1} 2022-03-06 11:42:28,123 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646566048120","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900003,"client":"172.30.41.118:42680","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-06 11:42:28,123 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646566048120","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900003,"client":"172.30.41.155:57328","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-06 11:42:58,136 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646566077786","responsesize":"21487","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 13 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900350,"client":"172.31.0.219:47928","multi.service_calls":0,"queuetimems":29666,"class":"HRegionServer","multi.mutations":13} 2022-03-06 11:43:14,508 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2641ms No GCs detected 2022-03-06 11:43:16,089 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1081ms No GCs detected 2022-03-06 11:43:25,176 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 8586ms No GCs detected 2022-03-06 11:43:27,419 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1181ms No GCs detected 2022-03-06 11:43:41,340 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1426ms No GCs detected 2022-03-06 11:43:53,431 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2083ms No GCs detected 2022-03-06 11:44:00,242 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: java.net.ConnectException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 11:46:18,275 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2914ms No GCs detected 2022-03-06 11:46:31,292 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 11:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=94536, hits=86437, hitRatio=91.43%, , cachingAccesses=86556, cachingHits=86128, cachingHitsRatio=99.51%, evictions=11558, evicted=0, evictedPerRun=0.0 2022-03-06 11:46:48,074 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 11:47:01,966 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646566621964","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:42766","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 11:47:05,229 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646566925229","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:57422","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 11:47:28,124 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646566948123","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:42812","multi.service_calls":0,"queuetimems":22894,"class":"HRegionServer","multi.mutations":1} 2022-03-06 11:47:28,124 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646566948123","responsesize":"9917","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:42812","multi.service_calls":0,"queuetimems":22894,"class":"HRegionServer","multi.mutations":6} 2022-03-06 11:47:28,195 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646566648193","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.31.0.219:47964","multi.service_calls":0,"queuetimems":26228,"class":"HRegionServer","multi.mutations":4} 2022-03-06 11:47:55,473 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646566675472","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:42776","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-06 11:47:58,137 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646566978136","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:57422","multi.service_calls":0,"queuetimems":30012,"class":"HRegionServer","multi.mutations":1} 2022-03-06 11:47:58,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646566377739","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 8 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900399,"client":"172.30.41.118:42730","multi.service_calls":0,"queuetimems":2268,"class":"HRegionServer","multi.mutations":8} 2022-03-06 11:47:59,062 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646566379060","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 7 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.155:57366","multi.service_calls":0,"queuetimems":3589,"class":"HRegionServer","multi.mutations":7} 2022-03-06 11:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=95334, hits=87235, hitRatio=91.50%, , cachingAccesses=87354, cachingHits=86926, cachingHitsRatio=99.51%, evictions=11588, evicted=0, evictedPerRun=0.0 2022-03-06 11:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 11:52:01,967 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646567221967","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:42862","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-06 11:52:05,231 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646567225230","responsesize":"9917","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:42862","multi.service_calls":0,"queuetimems":3263,"class":"HRegionServer","multi.mutations":6} 2022-03-06 11:52:05,243 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646566625242","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 5 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:47964","multi.service_calls":0,"queuetimems":3277,"class":"HRegionServer","multi.mutations":5} 2022-03-06 11:52:28,194 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646566048192","responsesize":"14885","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 9 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1500002,"client":"172.30.41.155:57328","multi.service_calls":0,"queuetimems":72,"class":"HRegionServer","multi.mutations":9} 2022-03-06 11:52:37,753 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646566057750","responsesize":"26449","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 16 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1500003,"client":"172.30.41.118:42680","multi.service_calls":0,"queuetimems":9630,"class":"HRegionServer","multi.mutations":16} 2022-03-06 11:52:58,137 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646566677790","responsesize":"14879","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 9 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900346,"client":"172.30.41.118:42776","multi.service_calls":0,"queuetimems":2318,"class":"HRegionServer","multi.mutations":9} 2022-03-06 11:52:58,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646567278138","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:57466","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-06 11:52:59,063 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646567279062","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:48008","multi.service_calls":0,"queuetimems":924,"class":"HRegionServer","multi.mutations":1} 2022-03-06 11:53:25,179 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646566700389","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":904790,"client":"172.30.41.155:57398","multi.service_calls":0,"queuetimems":24917,"class":"HRegionServer","multi.mutations":3} 2022-03-06 11:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=96102, hits=88003, hitRatio=91.57%, , cachingAccesses=88122, cachingHits=87694, cachingHitsRatio=99.51%, evictions=11618, evicted=0, evictedPerRun=0.0 2022-03-06 11:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 11:57:01,969 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646567521969","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:42912","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-06 11:57:05,232 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646567525231","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:57496","multi.service_calls":0,"queuetimems":3263,"class":"HRegionServer","multi.mutations":1} 2022-03-06 11:57:05,244 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646567525243","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:42912","multi.service_calls":0,"queuetimems":3275,"class":"HRegionServer","multi.mutations":1} 2022-03-06 11:57:28,125 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646567248124","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:48002","multi.service_calls":0,"queuetimems":26157,"class":"HRegionServer","multi.mutations":2} 2022-03-06 11:57:28,195 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646567548194","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:42912","multi.service_calls":0,"queuetimems":26226,"class":"HRegionServer","multi.mutations":1} 2022-03-06 11:57:28,195 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646566648193","responsesize":"13230","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 8 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200002,"client":"172.30.41.155:57390","multi.service_calls":0,"queuetimems":26228,"class":"HRegionServer","multi.mutations":8} 2022-03-06 11:57:28,195 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646566648193","responsesize":"21490","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 13 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.118:42766","multi.service_calls":0,"queuetimems":26228,"class":"HRegionServer","multi.mutations":13} 2022-03-06 11:57:28,196 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646567248196","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 6 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:57460","multi.service_calls":0,"queuetimems":22966,"class":"HRegionServer","multi.mutations":6} 2022-03-06 11:57:55,475 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646567275474","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 6 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:57466","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":6} 2022-03-06 11:59:40,354 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2346ms No GCs detected 2022-03-06 12:00:24,594 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4061ms No GCs detected 2022-03-06 12:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=96969, hits=88870, hitRatio=91.65%, , cachingAccesses=88989, cachingHits=88561, cachingHitsRatio=99.52%, evictions=11648, evicted=0, evictedPerRun=0.0 2022-03-06 12:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 12:02:01,971 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646567821970","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:57534","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 12:02:05,245 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646567825244","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:57534","multi.service_calls":0,"queuetimems":3273,"class":"HRegionServer","multi.mutations":1} 2022-03-06 12:02:28,125 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646567248124","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 5 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:57460","multi.service_calls":0,"queuetimems":26157,"class":"HRegionServer","multi.mutations":5} 2022-03-06 12:02:28,196 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646567848195","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:57534","multi.service_calls":0,"queuetimems":26224,"class":"HRegionServer","multi.mutations":1} 2022-03-06 12:02:28,196 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646567848195","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:57534","multi.service_calls":0,"queuetimems":26224,"class":"HRegionServer","multi.mutations":1} 2022-03-06 12:02:37,755 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646567557754","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:57504","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 12:02:55,477 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646567875476","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:57542","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 12:02:58,139 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646567278137","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 8 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.118:42876","multi.service_calls":0,"queuetimems":2663,"class":"HRegionServer","multi.mutations":8} 2022-03-06 12:02:58,139 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646567578139","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 5 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:57504","multi.service_calls":0,"queuetimems":20385,"class":"HRegionServer","multi.mutations":5} 2022-03-06 12:03:25,181 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646567605180","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.31.0.219:48026","multi.service_calls":0,"queuetimems":47426,"class":"HRegionServer","multi.mutations":2} 2022-03-06 12:06:07,110 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1846ms No GCs detected 2022-03-06 12:06:35,060 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2154ms No GCs detected 2022-03-06 12:06:44,489 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=97747, hits=89648, hitRatio=91.71%, , cachingAccesses=89767, cachingHits=89339, cachingHitsRatio=99.52%, evictions=11677, evicted=0, evictedPerRun=0.0 2022-03-06 12:06:44,500 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 8939ms No GCs detected 2022-03-06 12:06:44,563 WARN [regionserver/cvp328:16201] util.Sleeper: We slept 17087ms instead of 5000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired 2022-03-06 12:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 12:07:05,233 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646567825232","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:42974","multi.service_calls":0,"queuetimems":3262,"class":"HRegionServer","multi.mutations":4} 2022-03-06 12:07:16,207 INFO [regionserver/cvp328:16201.Chore.1] hbase.ScheduledChore: Chore: MemstoreFlusherChore missed its start time 2022-03-06 12:07:16,207 INFO [regionserver/cvp328:16201.Chore.1] hbase.ScheduledChore: Chore: CompactionChecker missed its start time 2022-03-06 12:07:16,208 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 6390ms No GCs detected 2022-03-06 12:07:28,126 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646567848125","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:48042","multi.service_calls":0,"queuetimems":26154,"class":"HRegionServer","multi.mutations":3} 2022-03-06 12:07:28,196 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646568148196","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:57564","multi.service_calls":0,"queuetimems":26224,"class":"HRegionServer","multi.mutations":1} 2022-03-06 12:07:28,196 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646567848195","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 8 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:42974","multi.service_calls":0,"queuetimems":26224,"class":"HRegionServer","multi.mutations":8} 2022-03-06 12:07:37,755 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646568157755","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:57564","multi.service_calls":0,"queuetimems":32509,"class":"HRegionServer","multi.mutations":3} 2022-03-06 12:07:54,614 INFO [regionserver/cvp328:16201.Chore.3] hbase.ScheduledChore: Chore: MemstoreFlusherChore missed its start time 2022-03-06 12:07:54,614 INFO [regionserver/cvp328:16201.Chore.3] hbase.ScheduledChore: Chore: CompactionChecker missed its start time 2022-03-06 12:07:54,718 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4732ms No GCs detected 2022-03-06 12:07:54,797 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 12:07:58,138 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646567578137","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:42922","multi.service_calls":0,"queuetimems":20383,"class":"HRegionServer","multi.mutations":4} 2022-03-06 12:07:58,141 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646568178139","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.118:43066","multi.service_calls":0,"queuetimems":2661,"class":"HRegionServer","multi.mutations":2} 2022-03-06 12:07:59,066 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646567579063","responsesize":"16531","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 10 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900003,"client":"172.30.41.118:42922","multi.service_calls":0,"queuetimems":21309,"class":"HRegionServer","multi.mutations":10} 2022-03-06 12:08:17,912 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1647ms No GCs detected 2022-03-06 12:08:25,050 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 12:08:33,902 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2722ms No GCs detected 2022-03-06 12:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=98446, hits=90347, hitRatio=91.77%, , cachingAccesses=90466, cachingHits=90038, cachingHitsRatio=99.53%, evictions=11707, evicted=0, evictedPerRun=0.0 2022-03-06 12:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 12:12:02,144 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646568121972","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600171,"client":"172.30.41.155:57564","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 12:12:05,235 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646568425235","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:43114","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 12:12:05,247 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646568125246","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:43046","multi.service_calls":0,"queuetimems":3274,"class":"HRegionServer","multi.mutations":3} 2022-03-06 12:12:28,158 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646568448157","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.30.41.118:43122","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 12:12:28,197 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646568148196","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.31.0.219:48060","multi.service_calls":0,"queuetimems":26223,"class":"HRegionServer","multi.mutations":5} 2022-03-06 12:12:28,197 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646567848196","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 4 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:48042","multi.service_calls":0,"queuetimems":26225,"class":"HRegionServer","multi.mutations":4} 2022-03-06 12:12:55,479 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646568175478","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:57574","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-06 12:12:58,139 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646568478138","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:48080","multi.service_calls":0,"queuetimems":29941,"class":"HRegionServer","multi.mutations":1} 2022-03-06 12:12:58,141 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646568178139","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.118:43066","multi.service_calls":0,"queuetimems":2661,"class":"HRegionServer","multi.mutations":4} 2022-03-06 12:12:58,141 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646568478141","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:48080","multi.service_calls":0,"queuetimems":29943,"class":"HRegionServer","multi.mutations":1} 2022-03-06 12:13:40,144 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1566ms No GCs detected 2022-03-06 12:13:42,512 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1868ms No GCs detected 2022-03-06 12:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=99204, hits=91105, hitRatio=91.84%, , cachingAccesses=91224, cachingHits=90796, cachingHitsRatio=99.53%, evictions=11737, evicted=0, evictedPerRun=0.0 2022-03-06 12:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 12:17:02,145 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646568722145","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:57616","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 12:17:28,127 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646568148125","responsesize":"18183","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 11 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900002,"client":"172.30.41.118:43046","multi.service_calls":0,"queuetimems":26153,"class":"HRegionServer","multi.mutations":11} 2022-03-06 12:17:28,197 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646568748197","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:43172","multi.service_calls":0,"queuetimems":26022,"class":"HRegionServer","multi.mutations":2} 2022-03-06 12:17:28,198 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646568748197","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:43174","multi.service_calls":0,"queuetimems":22961,"class":"HRegionServer","multi.mutations":2} 2022-03-06 12:17:28,198 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646568448197","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:43114","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-06 12:17:55,481 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646568775480","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:48098","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 12:17:58,140 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646568778140","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:57620","multi.service_calls":0,"queuetimems":2660,"class":"HRegionServer","multi.mutations":2} 2022-03-06 12:17:58,142 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646568778141","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:48098","multi.service_calls":0,"queuetimems":2661,"class":"HRegionServer","multi.mutations":1} 2022-03-06 12:17:59,107 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646568479106","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 4 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:48080","multi.service_calls":0,"queuetimems":53871,"class":"HRegionServer","multi.mutations":4} 2022-03-06 12:18:51,855 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1449ms No GCs detected 2022-03-06 12:19:48,556 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1297ms No GCs detected 2022-03-06 12:19:51,993 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2936ms No GCs detected 2022-03-06 12:21:31,990 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 12:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=99969, hits=91870, hitRatio=91.90%, , cachingAccesses=91989, cachingHits=91561, cachingHitsRatio=99.53%, evictions=11767, evicted=0, evictedPerRun=0.0 2022-03-06 12:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 12:22:02,148 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646569022147","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:57634","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 12:22:05,237 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646568725236","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:43174","multi.service_calls":0,"queuetimems":3091,"class":"HRegionServer","multi.mutations":2} 2022-03-06 12:22:05,249 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646568725247","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.155:57616","multi.service_calls":0,"queuetimems":3102,"class":"HRegionServer","multi.mutations":4} 2022-03-06 12:22:28,251 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646569048198","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300053,"client":"172.30.41.118:43250","multi.service_calls":0,"queuetimems":26051,"class":"HRegionServer","multi.mutations":1} 2022-03-06 12:22:28,255 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646569048198","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300057,"client":"172.30.41.118:43250","multi.service_calls":0,"queuetimems":26051,"class":"HRegionServer","multi.mutations":1} 2022-03-06 12:22:28,256 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646568448197","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900059,"client":"172.31.0.219:48080","multi.service_calls":0,"queuetimems":22962,"class":"HRegionServer","multi.mutations":4} 2022-03-06 12:22:28,284 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646569048127","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300156,"client":"172.30.41.118:43250","multi.service_calls":0,"queuetimems":25980,"class":"HRegionServer","multi.mutations":3} 2022-03-06 12:22:58,141 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646569078140","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:43250","multi.service_calls":0,"queuetimems":55993,"class":"HRegionServer","multi.mutations":1} 2022-03-06 12:22:58,143 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646569078142","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:43250","multi.service_calls":0,"queuetimems":55995,"class":"HRegionServer","multi.mutations":1} 2022-03-06 12:22:58,143 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646568778141","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 6 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.118:43190","multi.service_calls":0,"queuetimems":2661,"class":"HRegionServer","multi.mutations":6} 2022-03-06 12:23:25,183 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646568205181","responsesize":"11578","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 7 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200001,"client":"172.30.41.155:57574","multi.service_calls":0,"queuetimems":29703,"class":"HRegionServer","multi.mutations":7} 2022-03-06 12:24:29,005 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1324ms No GCs detected 2022-03-06 12:24:36,344 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 6315ms No GCs detected 2022-03-06 12:25:09,342 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 12:26:19,835 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 5261ms No GCs detected 2022-03-06 12:26:41,844 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 12:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=100718, hits=92619, hitRatio=91.96%, , cachingAccesses=92738, cachingHits=92310, cachingHitsRatio=99.54%, evictions=11796, evicted=0, evictedPerRun=0.0 2022-03-06 12:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 12:26:56,235 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3399ms No GCs detected 2022-03-06 12:27:02,149 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646569322149","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:57650","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 12:27:05,238 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646569325237","responsesize":"6613","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:57650","multi.service_calls":0,"queuetimems":3088,"class":"HRegionServer","multi.mutations":4} 2022-03-06 12:27:05,249 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646569325249","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:43320","multi.service_calls":0,"queuetimems":3070,"class":"HRegionServer","multi.mutations":2} 2022-03-06 12:27:28,251 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646568748158","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 8 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900093,"client":"172.30.41.118:43174","multi.service_calls":0,"queuetimems":26013,"class":"HRegionServer","multi.mutations":8} 2022-03-06 12:27:28,256 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646569348255","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:57650","multi.service_calls":0,"queuetimems":23017,"class":"HRegionServer","multi.mutations":2} 2022-03-06 12:27:37,757 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646568457755","responsesize":"21490","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 13 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.118:43114","multi.service_calls":0,"queuetimems":9558,"class":"HRegionServer","multi.mutations":13} 2022-03-06 12:27:55,482 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646569075481","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:48114","multi.service_calls":0,"queuetimems":53334,"class":"HRegionServer","multi.mutations":2} 2022-03-06 12:27:59,977 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646569079107","responsesize":"16528","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 10 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":600044,"client":"172.31.0.219:48114","multi.service_calls":0,"queuetimems":56960,"class":"HRegionServer","multi.mutations":10} 2022-03-06 12:28:03,996 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1840ms No GCs detected 2022-03-06 12:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=101399, hits=93300, hitRatio=92.01%, , cachingAccesses=93419, cachingHits=92991, cachingHitsRatio=99.54%, evictions=11826, evicted=0, evictedPerRun=0.0 2022-03-06 12:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-06 12:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 12:32:05,239 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646569625238","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:43386","multi.service_calls":0,"queuetimems":500,"class":"HRegionServer","multi.mutations":1} 2022-03-06 12:32:28,252 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646569648252","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:43386","multi.service_calls":0,"queuetimems":23013,"class":"HRegionServer","multi.mutations":3} 2022-03-06 12:32:28,252 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646569048197","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 5 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900055,"client":"172.30.41.155:57634","multi.service_calls":0,"queuetimems":26050,"class":"HRegionServer","multi.mutations":5} 2022-03-06 12:32:28,284 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646569348284","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.31.0.219:48130","multi.service_calls":0,"queuetimems":26135,"class":"HRegionServer","multi.mutations":2} 2022-03-06 12:32:37,759 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646569657758","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:43398","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 12:32:58,144 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646569378143","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 4 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:48134","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-06 12:32:58,144 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646569378144","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:57654","multi.service_calls":0,"queuetimems":2,"class":"HRegionServer","multi.mutations":3} 2022-03-06 12:33:25,184 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646569405183","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 5 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:57654","multi.service_calls":0,"queuetimems":27039,"class":"HRegionServer","multi.mutations":5} 2022-03-06 12:34:02,166 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4692ms No GCs detected 2022-03-06 12:34:07,867 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2593ms No GCs detected 2022-03-06 12:34:15,893 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 12:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=102094, hits=93995, hitRatio=92.07%, , cachingAccesses=94114, cachingHits=93686, cachingHitsRatio=99.55%, evictions=11856, evicted=0, evictedPerRun=0.0 2022-03-06 12:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 12:37:04,736 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646569624733","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600003,"client":"172.31.0.219:48152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":6} 2022-03-06 12:37:05,240 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646569925240","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300000,"client":"172.30.41.118:43434","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 12:37:05,250 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646569625249","responsesize":"11572","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 7 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:57672","multi.service_calls":0,"queuetimems":487,"class":"HRegionServer","multi.mutations":7} 2022-03-06 12:37:28,252 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646569348251","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:43318","multi.service_calls":0,"queuetimems":26102,"class":"HRegionServer","multi.mutations":3} 2022-03-06 12:37:28,253 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646569948253","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:43434","multi.service_calls":0,"queuetimems":23013,"class":"HRegionServer","multi.mutations":3} 2022-03-06 12:37:28,256 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646569648256","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:57672","multi.service_calls":0,"queuetimems":23017,"class":"HRegionServer","multi.mutations":2} 2022-03-06 12:37:28,257 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646569348256","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 6 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":900001,"client":"172.30.41.118:43318","multi.service_calls":0,"queuetimems":26107,"class":"HRegionServer","multi.mutations":6} 2022-03-06 12:37:37,791 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646569957790","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.31.0.219:48198","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 12:37:58,143 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646569378142","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":900001,"client":"172.30.41.118:43336","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":4} 2022-03-06 12:37:59,978 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646569679977","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 6 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:43398","multi.service_calls":0,"queuetimems":22219,"class":"HRegionServer","multi.mutations":6} 2022-03-06 12:38:01,821 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646569981821","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300000,"client":"172.30.41.118:43446","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 12:38:25,185 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646570005185","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:43448","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 12:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=102841, hits=94742, hitRatio=92.12%, , cachingAccesses=94861, cachingHits=94433, cachingHitsRatio=99.55%, evictions=11886, evicted=0, evictedPerRun=0.0 2022-03-06 12:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 12:42:04,738 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646570224737","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:43482","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 12:42:05,242 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646570225242","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:57710","multi.service_calls":0,"queuetimems":504,"class":"HRegionServer","multi.mutations":2} 2022-03-06 12:42:05,251 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646570225250","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:57710","multi.service_calls":0,"queuetimems":512,"class":"HRegionServer","multi.mutations":1} 2022-03-06 12:42:28,253 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646569948252","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:57690","multi.service_calls":0,"queuetimems":23012,"class":"HRegionServer","multi.mutations":3} 2022-03-06 12:42:28,253 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646570248253","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:43484","multi.service_calls":0,"queuetimems":23485,"class":"HRegionServer","multi.mutations":2} 2022-03-06 12:42:28,257 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646570248256","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:48228","multi.service_calls":0,"queuetimems":23518,"class":"HRegionServer","multi.mutations":1} 2022-03-06 12:42:28,257 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646570248257","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300000,"client":"172.30.41.118:43482","multi.service_calls":0,"queuetimems":23016,"class":"HRegionServer","multi.mutations":1} 2022-03-06 12:42:28,285 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646569948284","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:57690","multi.service_calls":0,"queuetimems":23044,"class":"HRegionServer","multi.mutations":2} 2022-03-06 12:42:37,792 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646570257791","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:57714","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 12:42:55,483 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646569675482","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:57672","multi.service_calls":0,"queuetimems":50243,"class":"HRegionServer","multi.mutations":3} 2022-03-06 12:42:58,143 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646570278143","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:43496","multi.service_calls":0,"queuetimems":20321,"class":"HRegionServer","multi.mutations":2} 2022-03-06 12:42:58,209 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646569978206","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600003,"client":"172.31.0.219:48192","multi.service_calls":0,"queuetimems":52966,"class":"HRegionServer","multi.mutations":2} 2022-03-06 12:43:01,983 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646570281821","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300162,"client":"172.30.41.118:43498","multi.service_calls":0,"queuetimems":1842,"class":"HRegionServer","multi.mutations":1} 2022-03-06 12:43:25,217 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646570305216","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.30.41.118:43500","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 12:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=103697, hits=95598, hitRatio=92.19%, , cachingAccesses=95717, cachingHits=95289, cachingHitsRatio=99.55%, evictions=11916, evicted=0, evictedPerRun=0.0 2022-03-06 12:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 12:47:04,739 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646570524739","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:43548","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 12:47:28,255 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646570248253","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.118:43482","multi.service_calls":0,"queuetimems":23516,"class":"HRegionServer","multi.mutations":4} 2022-03-06 12:47:28,286 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646570548285","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:48246","multi.service_calls":0,"queuetimems":23041,"class":"HRegionServer","multi.mutations":1} 2022-03-06 12:47:28,372 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646570548257","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300114,"client":"172.30.41.118:43548","multi.service_calls":0,"queuetimems":23014,"class":"HRegionServer","multi.mutations":1} 2022-03-06 12:47:37,794 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646570557793","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:43560","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 12:47:58,210 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646570578209","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.30.41.118:43558","multi.service_calls":0,"queuetimems":20386,"class":"HRegionServer","multi.mutations":2} 2022-03-06 12:48:00,001 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646570279979","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600022,"client":"172.30.41.118:43498","multi.service_calls":0,"queuetimems":22188,"class":"HRegionServer","multi.mutations":3} 2022-03-06 12:48:23,388 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 12:49:44,692 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1740ms No GCs detected 2022-03-06 12:49:58,653 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 12:50:44,248 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1268ms No GCs detected 2022-03-06 12:51:17,755 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1013ms No GCs detected 2022-03-06 12:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=104294, hits=96195, hitRatio=92.23%, , cachingAccesses=96314, cachingHits=95886, cachingHitsRatio=99.56%, evictions=11945, evicted=0, evictedPerRun=0.0 2022-03-06 12:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 12:52:05,244 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646570525243","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:57728","multi.service_calls":0,"queuetimems":504,"class":"HRegionServer","multi.mutations":3} 2022-03-06 12:52:05,253 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646570525251","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 6 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.155:57728","multi.service_calls":0,"queuetimems":512,"class":"HRegionServer","multi.mutations":6} 2022-03-06 12:52:28,256 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646570548253","responsesize":"11572","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 7 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600003,"client":"172.31.0.219:48246","multi.service_calls":0,"queuetimems":23514,"class":"HRegionServer","multi.mutations":7} 2022-03-06 12:52:28,286 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646570848286","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:43616","multi.service_calls":0,"queuetimems":21664,"class":"HRegionServer","multi.mutations":1} 2022-03-06 12:52:28,372 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646570548254","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 5 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600118,"client":"172.30.41.118:43548","multi.service_calls":0,"queuetimems":23515,"class":"HRegionServer","multi.mutations":5} 2022-03-06 12:52:28,372 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646570848372","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:43616","multi.service_calls":0,"queuetimems":20309,"class":"HRegionServer","multi.mutations":1} 2022-03-06 12:52:38,051 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646570857794","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300257,"client":"172.30.41.118:43616","multi.service_calls":0,"queuetimems":33054,"class":"HRegionServer","multi.mutations":2} 2022-03-06 12:52:55,485 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646570575484","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:57730","multi.service_calls":0,"queuetimems":17691,"class":"HRegionServer","multi.mutations":2} 2022-03-06 12:52:58,145 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646570578143","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 9 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.118:43560","multi.service_calls":0,"queuetimems":20350,"class":"HRegionServer","multi.mutations":9} 2022-03-06 12:52:58,212 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646570878210","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.31.0.219:48268","multi.service_calls":0,"queuetimems":48865,"class":"HRegionServer","multi.mutations":1} 2022-03-06 12:53:00,098 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646570880098","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:48268","multi.service_calls":0,"queuetimems":50373,"class":"HRegionServer","multi.mutations":1} 2022-03-06 12:53:02,086 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646570581984","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 5 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600102,"client":"172.30.41.155:57730","multi.service_calls":0,"queuetimems":24191,"class":"HRegionServer","multi.mutations":5} 2022-03-06 12:54:12,334 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 5438ms No GCs detected 2022-03-06 12:54:35,367 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 12:56:13,026 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1944ms No GCs detected 2022-03-06 12:56:34,343 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1212ms No GCs detected 2022-03-06 12:56:38,285 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2927ms No GCs detected 2022-03-06 12:56:41,233 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 12:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=104939, hits=96840, hitRatio=92.28%, , cachingAccesses=96959, cachingHits=96531, cachingHitsRatio=99.56%, evictions=11975, evicted=0, evictedPerRun=0.0 2022-03-06 12:56:48,084 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 12:57:04,741 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646570824740","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:43616","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-06 12:57:12,543 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 12:57:28,256 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646570848255","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:57746","multi.service_calls":0,"queuetimems":23515,"class":"HRegionServer","multi.mutations":4} 2022-03-06 12:57:28,287 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646571148287","responsesize":"11569","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 7 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:43690","multi.service_calls":0,"queuetimems":23042,"class":"HRegionServer","multi.mutations":7} 2022-03-06 12:57:28,372 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646571148372","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:48284","multi.service_calls":0,"queuetimems":23127,"class":"HRegionServer","multi.mutations":1} 2022-03-06 12:57:28,631 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646570548257","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 6 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900374,"client":"172.31.0.219:48246","multi.service_calls":0,"queuetimems":23517,"class":"HRegionServer","multi.mutations":6} 2022-03-06 12:57:42,811 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 12:57:48,578 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1749ms No GCs detected 2022-03-06 12:57:55,485 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646571175485","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:57758","multi.service_calls":0,"queuetimems":27198,"class":"HRegionServer","multi.mutations":1} 2022-03-06 12:57:57,072 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1122ms No GCs detected 2022-03-06 12:57:58,212 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646571178212","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:43690","multi.service_calls":0,"queuetimems":29925,"class":"HRegionServer","multi.mutations":1} 2022-03-06 12:58:19,299 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1448ms No GCs detected 2022-03-06 12:58:22,598 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1615ms No GCs detected 2022-03-06 12:58:25,218 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646570605217","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 7 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:48250","multi.service_calls":0,"queuetimems":47423,"class":"HRegionServer","multi.mutations":7} 2022-03-06 12:58:43,207 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1874ms No GCs detected 2022-03-06 13:00:15,821 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1187ms No GCs detected 2022-03-06 13:01:03,789 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1405ms No GCs detected 2022-03-06 13:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=105593, hits=97494, hitRatio=92.33%, , cachingAccesses=97613, cachingHits=97185, cachingHitsRatio=99.56%, evictions=12005, evicted=0, evictedPerRun=0.0 2022-03-06 13:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 13:02:04,743 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646571424742","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:43760","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:02:05,246 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646571125245","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:57758","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 13:02:05,255 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646571125253","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.118:43690","multi.service_calls":0,"queuetimems":8,"class":"HRegionServer","multi.mutations":5} 2022-03-06 13:02:19,102 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 13:02:19,104 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 484ms. 2022-03-06 13:02:21,448 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 2 failover attempts. Trying to failover after sleeping for 1116ms. 2022-03-06 13:02:22,566 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: tNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 3 failover attempts. Trying to failover after sleeping for 2125ms. 2022-03-06 13:02:28,257 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646571148256","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 6 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:48284","multi.service_calls":0,"queuetimems":23011,"class":"HRegionServer","multi.mutations":6} 2022-03-06 13:02:28,331 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646571448287","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300044,"client":"172.30.41.118:43768","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 13:02:28,417 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646571448416","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:57780","multi.service_calls":0,"queuetimems":124,"class":"HRegionServer","multi.mutations":2} 2022-03-06 13:02:28,419 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646571148372","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600047,"client":"172.30.41.155:57758","multi.service_calls":0,"queuetimems":23127,"class":"HRegionServer","multi.mutations":3} 2022-03-06 13:02:28,490 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646571448489","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300001,"client":"172.30.41.118:43770","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:02:38,172 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646571158171","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:57758","multi.service_calls":0,"queuetimems":9884,"class":"HRegionServer","multi.mutations":6} 2022-03-06 13:02:55,492 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646571475485","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300007,"client":"172.30.41.155:57780","multi.service_calls":0,"queuetimems":26339,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:02:58,146 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646571178145","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:48284","multi.service_calls":0,"queuetimems":29857,"class":"HRegionServer","multi.mutations":2} 2022-03-06 13:02:58,213 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646571478212","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:43760","multi.service_calls":0,"queuetimems":27766,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:03:00,100 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646571180099","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 6 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:43690","multi.service_calls":0,"queuetimems":31812,"class":"HRegionServer","multi.mutations":6} 2022-03-06 13:03:02,087 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646571182086","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 8 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:48284","multi.service_calls":0,"queuetimems":33798,"class":"HRegionServer","multi.mutations":8} 2022-03-06 13:03:25,219 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646571505218","responsesize":"6613","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:48302","multi.service_calls":0,"queuetimems":80475,"class":"HRegionServer","multi.mutations":4} 2022-03-06 13:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=106222, hits=98123, hitRatio=92.38%, , cachingAccesses=98242, cachingHits=97814, cachingHitsRatio=99.56%, evictions=12035, evicted=0, evictedPerRun=0.0 2022-03-06 13:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 13:07:28,257 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646571748257","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:57790","multi.service_calls":0,"queuetimems":23496,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:07:28,417 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646571748417","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:57790","multi.service_calls":0,"queuetimems":23656,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:07:28,632 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646571448631","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:43760","multi.service_calls":0,"queuetimems":23889,"class":"HRegionServer","multi.mutations":3} 2022-03-06 13:07:28,695 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646571748695","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:57790","multi.service_calls":0,"queuetimems":23933,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:07:28,695 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646571748694","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:57790","multi.service_calls":0,"queuetimems":23933,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:07:39,944 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646571758172","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":301772,"client":"172.31.0.219:48314","multi.service_calls":0,"queuetimems":32925,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:07:55,492 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646571775492","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:48314","multi.service_calls":0,"queuetimems":50245,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:07:58,291 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646571778213","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300077,"client":"172.31.0.219:48314","multi.service_calls":0,"queuetimems":52966,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:08:00,102 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646571780100","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.31.0.219:48314","multi.service_calls":0,"queuetimems":54853,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=106897, hits=98798, hitRatio=92.42%, , cachingAccesses=98917, cachingHits=98489, cachingHitsRatio=99.57%, evictions=12065, evicted=0, evictedPerRun=0.0 2022-03-06 13:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 13:12:04,745 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646571724744","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:43826","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-06 13:12:05,299 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646571725246","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600053,"client":"172.31.0.219:48314","multi.service_calls":0,"queuetimems":502,"class":"HRegionServer","multi.mutations":2} 2022-03-06 13:12:28,432 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646571748331","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 8 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600101,"client":"172.30.41.118:43826","multi.service_calls":0,"queuetimems":23587,"class":"HRegionServer","multi.mutations":8} 2022-03-06 13:12:28,433 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646572048258","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300175,"client":"172.30.41.118:43898","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:12:28,433 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646572048418","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300015,"client":"172.30.41.155:57810","multi.service_calls":0,"queuetimems":160,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:12:28,696 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646572048695","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:48326","multi.service_calls":0,"queuetimems":436,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:12:28,696 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646572048695","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:57810","multi.service_calls":0,"queuetimems":437,"class":"HRegionServer","multi.mutations":3} 2022-03-06 13:12:41,894 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646572061893","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:48326","multi.service_calls":0,"queuetimems":13634,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:12:55,493 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646572075492","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:48326","multi.service_calls":0,"queuetimems":27233,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:12:58,291 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646572078291","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:48326","multi.service_calls":0,"queuetimems":30032,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:13:00,103 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646572080102","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:43898","multi.service_calls":0,"queuetimems":31684,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:13:25,221 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646571805221","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.31.0.219:48318","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-06 13:15:18,460 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1663ms No GCs detected 2022-03-06 13:16:36,666 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 13:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=107597, hits=99498, hitRatio=92.47%, , cachingAccesses=99617, cachingHits=99189, cachingHitsRatio=99.57%, evictions=12095, evicted=0, evictedPerRun=0.0 2022-03-06 13:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 13:17:04,747 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646572324746","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:43956","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:17:05,257 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646571725255","responsesize":"18183","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 11 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900002,"client":"172.31.0.219:48314","multi.service_calls":0,"queuetimems":511,"class":"HRegionServer","multi.mutations":11} 2022-03-06 13:17:05,302 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646572325300","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:43956","multi.service_calls":0,"queuetimems":554,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:17:28,433 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646572348432","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:43956","multi.service_calls":0,"queuetimems":23686,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:17:28,433 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646572348433","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:43956","multi.service_calls":0,"queuetimems":23687,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:17:28,434 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646572348433","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:43956","multi.service_calls":0,"queuetimems":23687,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:17:28,633 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646572048632","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:43898","multi.service_calls":0,"queuetimems":374,"class":"HRegionServer","multi.mutations":3} 2022-03-06 13:17:28,696 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646572348696","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:43956","multi.service_calls":0,"queuetimems":23950,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:17:28,696 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646572348696","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300000,"client":"172.30.41.118:43956","multi.service_calls":0,"queuetimems":23950,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:17:41,895 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646572361894","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:43972","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:17:55,494 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646572375493","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:43972","multi.service_calls":0,"queuetimems":13598,"class":"HRegionServer","multi.mutations":2} 2022-03-06 13:17:58,298 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646572378297","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:57826","multi.service_calls":0,"queuetimems":16402,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:17:59,681 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646571778147","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":901534,"client":"172.30.41.118:43834","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":4} 2022-03-06 13:18:00,109 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646572380103","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300006,"client":"172.30.41.118:43972","multi.service_calls":0,"queuetimems":18208,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:18:02,165 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646571782087","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 8 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900078,"client":"172.30.41.118:43834","multi.service_calls":0,"queuetimems":3940,"class":"HRegionServer","multi.mutations":8} 2022-03-06 13:18:09,674 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1683ms No GCs detected 2022-03-06 13:18:25,222 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646572405222","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300000,"client":"172.30.41.118:43972","multi.service_calls":0,"queuetimems":43328,"class":"HRegionServer","multi.mutations":3} 2022-03-06 13:19:30,346 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 7313ms No GCs detected 2022-03-06 13:19:40,043 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2931ms No GCs detected 2022-03-06 13:20:14,718 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 13:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=108225, hits=100126, hitRatio=92.52%, , cachingAccesses=100245, cachingHits=99817, cachingHitsRatio=99.57%, evictions=12125, evicted=0, evictedPerRun=0.0 2022-03-06 13:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 13:22:04,748 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646572624748","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:44026","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:22:05,258 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646572625258","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:44034","multi.service_calls":0,"queuetimems":480,"class":"HRegionServer","multi.mutations":2} 2022-03-06 13:22:05,302 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646572625302","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:44026","multi.service_calls":0,"queuetimems":44,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:22:39,172 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646572659172","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:44036","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 13:22:58,299 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646572678298","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:48362","multi.service_calls":0,"queuetimems":624,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:23:00,110 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646572680110","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:48362","multi.service_calls":0,"queuetimems":428,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:23:02,166 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646572682165","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300001,"client":"172.30.41.118:44038","multi.service_calls":0,"queuetimems":2360,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:23:25,223 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646572705222","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:57844","multi.service_calls":0,"queuetimems":25108,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:25:59,718 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1274ms No GCs detected 2022-03-06 13:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=108923, hits=100824, hitRatio=92.56%, , cachingAccesses=100943, cachingHits=100515, cachingHitsRatio=99.58%, evictions=12155, evicted=0, evictedPerRun=0.0 2022-03-06 13:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 13:27:04,749 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646572924749","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:44074","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:27:05,303 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646572925302","responsesize":"11569","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:48374","multi.service_calls":0,"queuetimems":553,"class":"HRegionServer","multi.mutations":7} 2022-03-06 13:27:28,434 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646572648433","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:57842","multi.service_calls":0,"queuetimems":23685,"class":"HRegionServer","multi.mutations":4} 2022-03-06 13:27:28,434 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646572648433","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 5 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:44026","multi.service_calls":0,"queuetimems":23175,"class":"HRegionServer","multi.mutations":5} 2022-03-06 13:27:55,252 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2624ms No GCs detected 2022-03-06 13:27:55,512 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646572675494","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600018,"client":"172.31.0.219:48360","multi.service_calls":0,"queuetimems":50744,"class":"HRegionServer","multi.mutations":3} 2022-03-06 13:27:57,675 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646572677674","responsesize":"16528","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 10 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":600001,"client":"172.31.0.219:48362","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":10} 2022-03-06 13:27:59,715 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646572679713","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.118:44040","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-06 13:28:00,112 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646572980110","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.155:57874","multi.service_calls":0,"queuetimems":20936,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:28:02,468 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646572982166","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300302,"client":"172.30.41.118:44086","multi.service_calls":0,"queuetimems":3867,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:28:19,611 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 13:31:28,144 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 13:31:33,720 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1366ms No GCs detected 2022-03-06 13:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=109633, hits=101534, hitRatio=92.61%, , cachingAccesses=101653, cachingHits=101225, cachingHitsRatio=99.58%, evictions=12185, evicted=0, evictedPerRun=0.0 2022-03-06 13:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-06 13:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 13:31:59,584 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3056ms No GCs detected 2022-03-06 13:32:00,573 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 13:32:05,260 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646572925259","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:57870","multi.service_calls":0,"queuetimems":510,"class":"HRegionServer","multi.mutations":4} 2022-03-06 13:32:05,304 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646573225303","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:57902","multi.service_calls":0,"queuetimems":553,"class":"HRegionServer","multi.mutations":3} 2022-03-06 13:32:10,639 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646573224750","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":305889,"client":"172.30.41.155:57902","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:32:10,658 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 768ms. 2022-03-06 13:32:11,436 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: tNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 2 failover attempts. Trying to failover after sleeping for 1698ms. 2022-03-06 13:32:28,435 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646572648434","responsesize":"19835","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 12 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Dev ","processingtimems":900001,"client":"172.30.41.118:44026","multi.service_calls":0,"queuetimems":23686,"class":"HRegionServer","multi.mutations":12} 2022-03-06 13:32:28,549 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646573248434","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300115,"client":"172.31.0.219:48394","multi.service_calls":0,"queuetimems":23683,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:32:41,295 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3274ms No GCs detected 2022-03-06 13:32:43,526 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 13:32:58,074 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646573278074","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:57910","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:32:58,300 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646572978299","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:44086","multi.service_calls":0,"queuetimems":19126,"class":"HRegionServer","multi.mutations":4} 2022-03-06 13:32:59,717 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646573279715","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300002,"client":"172.30.41.155:57908","multi.service_calls":0,"queuetimems":416,"class":"HRegionServer","multi.mutations":2} 2022-03-06 13:33:00,114 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646573280113","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:48396","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:33:14,846 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 13:33:25,224 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646573005223","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:57874","multi.service_calls":0,"queuetimems":26923,"class":"HRegionServer","multi.mutations":2} 2022-03-06 13:33:36,999 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1147ms No GCs detected 2022-03-06 13:33:39,663 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2164ms No GCs detected 2022-03-06 13:34:23,763 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1448ms No GCs detected 2022-03-06 13:34:33,256 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1570ms No GCs detected 2022-03-06 13:34:37,677 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1001ms No GCs detected 2022-03-06 13:34:48,683 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 13:35:28,244 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 13:36:29,208 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1833ms No GCs detected 2022-03-06 13:36:33,174 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1044ms No GCs detected 2022-03-06 13:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=110384, hits=102285, hitRatio=92.66%, , cachingAccesses=102404, cachingHits=101976, cachingHitsRatio=99.58%, evictions=12215, evicted=0, evictedPerRun=0.0 2022-03-06 13:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 13:37:04,211 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 13:37:07,103 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646573527102","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:48410","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:37:07,103 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646573527102","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:48410","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 13:37:10,640 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646573530639","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:44182","multi.service_calls":0,"queuetimems":3401,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:37:28,435 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646573548435","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:44182","multi.service_calls":0,"queuetimems":21197,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:37:28,549 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646573548549","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:44182","multi.service_calls":0,"queuetimems":21443,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:37:28,698 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646572648696","responsesize":"9926","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 6 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200002,"client":"172.30.41.155:57842","multi.service_calls":0,"queuetimems":23948,"class":"HRegionServer","multi.mutations":6} 2022-03-06 13:37:28,840 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646572648696","responsesize":"21490","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 13 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200144,"client":"172.31.0.219:48360","multi.service_calls":0,"queuetimems":23946,"class":"HRegionServer","multi.mutations":13} 2022-03-06 13:37:39,787 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646572959173","responsesize":"14879","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 9 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900614,"client":"172.30.41.118:44086","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":9} 2022-03-06 13:37:50,868 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2769ms No GCs detected 2022-03-06 13:37:58,075 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646573578075","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:57926","multi.service_calls":0,"queuetimems":50767,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:37:58,302 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646573578300","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.155:57926","multi.service_calls":0,"queuetimems":50992,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:37:59,717 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646573579717","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:57926","multi.service_calls":0,"queuetimems":52409,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:38:00,115 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646573580114","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:57926","multi.service_calls":0,"queuetimems":52806,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:38:05,546 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 13:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=111123, hits=103024, hitRatio=92.71%, , cachingAccesses=103143, cachingHits=102715, cachingHitsRatio=99.59%, evictions=12244, evicted=0, evictedPerRun=0.0 2022-03-06 13:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 13:42:07,104 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646573827104","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:44258","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:42:29,405 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646573248434","responsesize":"16531","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 10 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900971,"client":"172.30.41.118:44130","multi.service_calls":0,"queuetimems":23684,"class":"HRegionServer","multi.mutations":10} 2022-03-06 13:42:29,406 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646573848550","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300856,"client":"172.30.41.155:57942","multi.service_calls":0,"queuetimems":21446,"class":"HRegionServer","multi.mutations":2} 2022-03-06 13:42:29,406 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1318ms No GCs detected 2022-03-06 13:42:55,513 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646573275512","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 6 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:44130","multi.service_calls":0,"queuetimems":50762,"class":"HRegionServer","multi.mutations":6} 2022-03-06 13:42:58,076 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646573878075","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:48432","multi.service_calls":0,"queuetimems":47434,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:42:58,302 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646573878302","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:48432","multi.service_calls":0,"queuetimems":47661,"class":"HRegionServer","multi.mutations":2} 2022-03-06 13:43:00,129 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646573880116","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300013,"client":"172.30.41.118:44266","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:43:03,911 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1036ms No GCs detected 2022-03-06 13:46:01,523 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1098ms No GCs detected 2022-03-06 13:46:03,145 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1121ms No GCs detected 2022-03-06 13:46:13,947 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1295ms No GCs detected 2022-03-06 13:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=111832, hits=103733, hitRatio=92.76%, , cachingAccesses=103852, cachingHits=103424, cachingHitsRatio=99.59%, evictions=12274, evicted=0, evictedPerRun=0.0 2022-03-06 13:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 13:47:07,105 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646573827104","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 5 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:44258","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":5} 2022-03-06 13:47:07,106 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646574127106","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:44316","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":2} 2022-03-06 13:47:10,642 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646573830640","responsesize":"11572","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 7 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.31.0.219:48432","multi.service_calls":0,"queuetimems":3535,"class":"HRegionServer","multi.mutations":7} 2022-03-06 13:47:29,405 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646573848435","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600970,"client":"172.30.41.155:57942","multi.service_calls":0,"queuetimems":17794,"class":"HRegionServer","multi.mutations":2} 2022-03-06 13:47:29,406 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646574149406","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:57958","multi.service_calls":0,"queuetimems":22301,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:47:29,406 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646574149406","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:57958","multi.service_calls":0,"queuetimems":22300,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:47:29,406 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646573848698","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600708,"client":"172.30.41.118:44258","multi.service_calls":0,"queuetimems":18057,"class":"HRegionServer","multi.mutations":3} 2022-03-06 13:47:39,788 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646573859787","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:48432","multi.service_calls":0,"queuetimems":32682,"class":"HRegionServer","multi.mutations":4} 2022-03-06 13:47:55,514 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646574175513","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:57958","multi.service_calls":0,"queuetimems":48407,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:47:59,724 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646573879718","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600006,"client":"172.31.0.219:48432","multi.service_calls":0,"queuetimems":49077,"class":"HRegionServer","multi.mutations":3} 2022-03-06 13:48:00,130 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646574180129","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:57958","multi.service_calls":0,"queuetimems":53023,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:48:02,470 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646573282468","responsesize":"21490","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 13 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.31.0.219:48396","multi.service_calls":0,"queuetimems":2355,"class":"HRegionServer","multi.mutations":13} 2022-03-06 13:48:18,554 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1281ms No GCs detected 2022-03-06 13:48:25,226 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646573605225","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 8 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:44204","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":8} 2022-03-06 13:48:31,646 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1017ms No GCs detected 2022-03-06 13:48:44,733 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2513ms No GCs detected 2022-03-06 13:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=112657, hits=104558, hitRatio=92.81%, , cachingAccesses=104677, cachingHits=104249, cachingHitsRatio=99.59%, evictions=12304, evicted=0, evictedPerRun=0.0 2022-03-06 13:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 13:52:29,407 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646574449406","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:48466","multi.service_calls":0,"queuetimems":22300,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:52:55,515 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646574475515","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:44408","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:52:58,077 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646574178076","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:44316","multi.service_calls":0,"queuetimems":50970,"class":"HRegionServer","multi.mutations":6} 2022-03-06 13:52:58,305 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646574178303","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.118:44316","multi.service_calls":0,"queuetimems":51197,"class":"HRegionServer","multi.mutations":4} 2022-03-06 13:53:00,131 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646574480130","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:48468","multi.service_calls":0,"queuetimems":4615,"class":"HRegionServer","multi.mutations":2} 2022-03-06 13:53:02,470 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646574482470","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:57978","multi.service_calls":0,"queuetimems":6864,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:53:25,227 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646574505226","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:44408","multi.service_calls":0,"queuetimems":25501,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:53:43,477 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1715ms No GCs detected 2022-03-06 13:53:48,087 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3609ms No GCs detected 2022-03-06 13:53:50,819 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2231ms No GCs detected 2022-03-06 13:54:28,313 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1516ms No GCs detected 2022-03-06 13:54:35,967 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3144ms No GCs detected 2022-03-06 13:54:52,658 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 13:55:24,810 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 13:55:28,685 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1097ms No GCs detected 2022-03-06 13:55:28,939 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 420ms. 2022-03-06 13:56:29,543 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 13:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=113540, hits=105441, hitRatio=92.87%, , cachingAccesses=105560, cachingHits=105132, cachingHitsRatio=99.59%, evictions=12334, evicted=0, evictedPerRun=0.0 2022-03-06 13:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 13:57:07,122 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646574427106","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600016,"client":"172.30.41.118:44400","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 13:57:07,213 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646574427106","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600107,"client":"172.30.41.155:57976","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-06 13:57:29,406 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646573848840","responsesize":"13230","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 8 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200566,"client":"172.30.41.155:57942","multi.service_calls":0,"queuetimems":18199,"class":"HRegionServer","multi.mutations":8} 2022-03-06 13:57:29,407 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646574449406","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:44400","multi.service_calls":0,"queuetimems":18762,"class":"HRegionServer","multi.mutations":2} 2022-03-06 13:57:29,407 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646574449406","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:48466","multi.service_calls":0,"queuetimems":18762,"class":"HRegionServer","multi.mutations":2} 2022-03-06 13:57:39,789 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646574459788","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:44400","multi.service_calls":0,"queuetimems":29144,"class":"HRegionServer","multi.mutations":3} 2022-03-06 13:57:58,508 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3567ms No GCs detected 2022-03-06 13:57:58,547 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646574778305","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300242,"client":"172.31.0.219:48486","multi.service_calls":0,"queuetimems":28897,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:58:00,131 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646574780131","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:48486","multi.service_calls":0,"queuetimems":30723,"class":"HRegionServer","multi.mutations":1} 2022-03-06 13:58:00,512 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 13:58:25,228 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646574805227","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:48486","multi.service_calls":0,"queuetimems":55819,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:00:56,342 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1081ms No GCs detected 2022-03-06 14:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=114370, hits=106271, hitRatio=92.92%, , cachingAccesses=106390, cachingHits=105962, cachingHitsRatio=99.60%, evictions=12364, evicted=0, evictedPerRun=0.0 2022-03-06 14:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 14:02:07,124 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646575027123","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:44548","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:02:07,217 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646575027217","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:48502","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:02:10,645 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646574430643","responsesize":"19835","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 12 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Dev ","processingtimems":900002,"client":"172.30.41.118:44400","multi.service_calls":0,"queuetimems":3537,"class":"HRegionServer","multi.mutations":12} 2022-03-06 14:02:29,528 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646574449405","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 7 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900123,"client":"172.30.41.155:57976","multi.service_calls":0,"queuetimems":22299,"class":"HRegionServer","multi.mutations":7} 2022-03-06 14:02:34,745 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 14:02:35,650 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1410ms No GCs detected 2022-03-06 14:02:37,494 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 517ms. 2022-03-06 14:02:39,790 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646575059789","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:58042","multi.service_calls":0,"queuetimems":10383,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:02:58,548 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646575078547","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:44548","multi.service_calls":0,"queuetimems":29060,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:02:59,741 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646574479724","responsesize":"19835","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 12 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900016,"client":"172.31.0.219:48468","multi.service_calls":0,"queuetimems":4209,"class":"HRegionServer","multi.mutations":12} 2022-03-06 14:03:00,181 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646575080131","responsesize":"9917","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300050,"client":"172.30.41.118:44548","multi.service_calls":0,"queuetimems":30644,"class":"HRegionServer","multi.mutations":6} 2022-03-06 14:03:03,024 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646574782470","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600554,"client":"172.30.41.155:58008","multi.service_calls":0,"queuetimems":33062,"class":"HRegionServer","multi.mutations":2} 2022-03-06 14:03:08,189 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 14:03:29,645 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1983ms No GCs detected 2022-03-06 14:03:32,134 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1055ms GC pool 'G1 Young Generation' had collection(s): count=1 time=1386ms 2022-03-06 14:03:35,579 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1617ms No GCs detected 2022-03-06 14:04:14,165 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4116ms No GCs detected 2022-03-06 14:04:31,934 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1885ms GC pool 'G1 Young Generation' had collection(s): count=1 time=2379ms 2022-03-06 14:04:43,937 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 14:04:57,084 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2491ms No GCs detected 2022-03-06 14:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=115063, hits=106964, hitRatio=92.96%, , cachingAccesses=107083, cachingHits=106655, cachingHitsRatio=99.60%, evictions=12394, evicted=0, evictedPerRun=0.0 2022-03-06 14:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 14:07:07,149 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646575327125","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300024,"client":"172.30.41.118:44602","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:07:10,647 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646575330646","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:58072","multi.service_calls":0,"queuetimems":3521,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:07:29,487 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646574749408","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900079,"client":"172.30.41.118:44480","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-06 14:07:39,790 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646575359790","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:48518","multi.service_calls":0,"queuetimems":32663,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:07:58,098 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646574778077","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 6 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900021,"client":"172.30.41.155:58008","multi.service_calls":0,"queuetimems":28669,"class":"HRegionServer","multi.mutations":6} 2022-03-06 14:07:58,548 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646575378548","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:44602","multi.service_calls":0,"queuetimems":47902,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:07:58,688 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646574775516","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 7 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":903172,"client":"172.30.41.118:44480","multi.service_calls":0,"queuetimems":26108,"class":"HRegionServer","multi.mutations":7} 2022-03-06 14:08:00,182 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646575380181","responsesize":"9917","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:44602","multi.service_calls":0,"queuetimems":49535,"class":"HRegionServer","multi.mutations":6} 2022-03-06 14:08:03,026 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646575383026","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:48520","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:08:27,166 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646575105228","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":601938,"client":"172.30.41.118:44556","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 14:08:46,627 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 14:10:03,384 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1218ms No GCs detected 2022-03-06 14:10:36,685 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1176ms No GCs detected 2022-03-06 14:11:04,783 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 5664ms No GCs detected 2022-03-06 14:11:17,869 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 14:11:24,692 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2006ms No GCs detected 2022-03-06 14:11:42,049 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1375ms No GCs detected 2022-03-06 14:11:44,496 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=115837, hits=107738, hitRatio=93.01%, , cachingAccesses=107857, cachingHits=107429, cachingHitsRatio=99.60%, evictions=12423, evicted=0, evictedPerRun=0.0 2022-03-06 14:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 14:12:07,174 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646575627150","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300024,"client":"172.30.41.155:58094","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:12:10,648 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646575630648","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:44670","multi.service_calls":0,"queuetimems":3498,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:12:18,700 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 14:12:29,488 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646575649487","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:58094","multi.service_calls":0,"queuetimems":22337,"class":"HRegionServer","multi.mutations":3} 2022-03-06 14:12:29,529 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646575049407","responsesize":"19835","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 12 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900122,"client":"172.31.0.219:48502","multi.service_calls":0,"queuetimems":22190,"class":"HRegionServer","multi.mutations":12} 2022-03-06 14:12:29,529 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646575049407","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 8 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900122,"client":"172.30.41.155:58042","multi.service_calls":0,"queuetimems":22254,"class":"HRegionServer","multi.mutations":8} 2022-03-06 14:12:39,791 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646575659791","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:44670","multi.service_calls":0,"queuetimems":32641,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:12:58,099 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646575678098","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:44670","multi.service_calls":0,"queuetimems":50948,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:12:58,689 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646575678688","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:48536","multi.service_calls":0,"queuetimems":48040,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:12:59,742 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646575379741","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:58072","multi.service_calls":0,"queuetimems":49095,"class":"HRegionServer","multi.mutations":3} 2022-03-06 14:13:03,027 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646575683026","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:44670","multi.service_calls":0,"queuetimems":55876,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:13:07,757 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646575687757","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:58096","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 14:13:27,167 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646575707167","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:58096","multi.service_calls":0,"queuetimems":19410,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:14:03,293 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1178ms No GCs detected 2022-03-06 14:14:40,818 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1887ms No GCs detected 2022-03-06 14:15:59,253 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 14:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=116616, hits=108517, hitRatio=93.05%, , cachingAccesses=108636, cachingHits=108208, cachingHitsRatio=99.61%, evictions=12453, evicted=0, evictedPerRun=0.0 2022-03-06 14:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 14:17:07,640 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646575927505","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300135,"client":"172.31.0.219:48558","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:17:08,205 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646575327217","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 8 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900988,"client":"172.30.41.155:58072","multi.service_calls":0,"queuetimems":92,"class":"HRegionServer","multi.mutations":8} 2022-03-06 14:17:07,640 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1085ms No GCs detected 2022-03-06 14:17:10,678 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646575930649","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300029,"client":"172.30.41.155:58110","multi.service_calls":0,"queuetimems":2491,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:17:30,154 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646575949529","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300625,"client":"172.30.41.155:58110","multi.service_calls":0,"queuetimems":18880,"class":"HRegionServer","multi.mutations":2} 2022-03-06 14:17:37,008 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646575949488","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":307520,"client":"172.31.0.219:48558","multi.service_calls":0,"queuetimems":21983,"class":"HRegionServer","multi.mutations":2} 2022-03-06 14:17:37,040 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646575949529","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":307511,"client":"172.30.41.118:44736","multi.service_calls":0,"queuetimems":21369,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:17:37,041 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646575049406","responsesize":"18186","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 11 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1207634,"client":"172.30.41.118:44548","multi.service_calls":0,"queuetimems":22283,"class":"HRegionServer","multi.mutations":11} 2022-03-06 14:17:48,114 WARN [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 10853ms No GCs detected 2022-03-06 14:17:48,212 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646575959791","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":308421,"client":"172.30.41.155:58110","multi.service_calls":0,"queuetimems":31903,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:17:54,266 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 14:17:58,100 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646575978100","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:44748","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:17:58,720 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646575978689","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300031,"client":"172.30.41.118:44738","multi.service_calls":0,"queuetimems":29103,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:18:00,128 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646575979742","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300385,"client":"172.31.0.219:48560","multi.service_calls":0,"queuetimems":1642,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:18:03,566 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646575983047","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300519,"client":"172.30.41.118:44752","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:18:03,567 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1064ms No GCs detected 2022-03-06 14:18:27,168 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646576007167","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:44748","multi.service_calls":0,"queuetimems":19409,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:18:38,979 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1316ms No GCs detected 2022-03-06 14:19:55,530 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 14:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=117232, hits=109133, hitRatio=93.09%, , cachingAccesses=109252, cachingHits=108824, cachingHitsRatio=99.61%, evictions=12482, evicted=0, evictedPerRun=0.0 2022-03-06 14:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 14:22:08,481 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646576228480","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:48596","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:22:08,481 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646576228480","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:48596","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:22:10,679 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646576230678","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:48596","multi.service_calls":0,"queuetimems":2198,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:22:15,449 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1120ms No GCs detected 2022-03-06 14:22:30,155 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646576250154","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:48596","multi.service_calls":0,"queuetimems":21673,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:22:30,451 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 14:22:30,453 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 554ms. 2022-03-06 14:22:37,009 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646576257008","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:48596","multi.service_calls":0,"queuetimems":28527,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:22:37,009 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646575349528","responsesize":"18186","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 11 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1207481,"client":"172.30.41.118:44602","multi.service_calls":0,"queuetimems":22403,"class":"HRegionServer","multi.mutations":11} 2022-03-06 14:22:37,041 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646576257040","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:48596","multi.service_calls":0,"queuetimems":28559,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:22:37,042 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646576257041","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:48596","multi.service_calls":0,"queuetimems":28560,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:22:48,223 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646576268222","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:48596","multi.service_calls":0,"queuetimems":39741,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:22:58,101 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646576278100","responsesize":"11569","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:48596","multi.service_calls":0,"queuetimems":46776,"class":"HRegionServer","multi.mutations":7} 2022-03-06 14:22:58,636 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646575678548","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900088,"client":"172.30.41.155:58094","multi.service_calls":0,"queuetimems":47900,"class":"HRegionServer","multi.mutations":7} 2022-03-06 14:22:58,721 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646576278721","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:48596","multi.service_calls":0,"queuetimems":50240,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:23:00,128 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646576280128","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:44800","multi.service_calls":0,"queuetimems":2027,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:23:02,772 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 14:23:04,475 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646576284475","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:58128","multi.service_calls":0,"queuetimems":2073,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:23:07,760 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646575987758","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:44748","multi.service_calls":0,"queuetimems":9658,"class":"HRegionServer","multi.mutations":4} 2022-03-06 14:23:27,223 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646576307222","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300000,"client":"172.30.41.118:44806","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=118022, hits=109923, hitRatio=93.14%, , cachingAccesses=110042, cachingHits=109614, cachingHitsRatio=99.61%, evictions=12512, evicted=0, evictedPerRun=0.0 2022-03-06 14:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 14:27:03,845 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1172ms No GCs detected 2022-03-06 14:27:08,482 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646576528482","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:58144","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:27:08,482 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646576528482","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:58144","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:27:10,680 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646576530679","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:44848","multi.service_calls":0,"queuetimems":2197,"class":"HRegionServer","multi.mutations":3} 2022-03-06 14:27:30,802 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646576550155","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300647,"client":"172.30.41.155:58144","multi.service_calls":0,"queuetimems":21673,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:27:37,009 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646576557009","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:58144","multi.service_calls":0,"queuetimems":28527,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:27:37,010 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646576557009","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:58144","multi.service_calls":0,"queuetimems":28527,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:27:37,043 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646576557042","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:58144","multi.service_calls":0,"queuetimems":28560,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:27:37,043 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646576557042","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:58144","multi.service_calls":0,"queuetimems":28560,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:27:58,102 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646576578101","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:58144","multi.service_calls":0,"queuetimems":47421,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:27:58,721 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646576578721","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:58144","multi.service_calls":0,"queuetimems":48041,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:28:00,133 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646576580128","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300004,"client":"172.30.41.155:58144","multi.service_calls":0,"queuetimems":49448,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:28:04,477 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646576584476","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:44856","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-06 14:28:07,760 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646576587760","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:44856","multi.service_calls":0,"queuetimems":3284,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:28:27,223 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646576607223","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:44856","multi.service_calls":0,"queuetimems":22747,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=118725, hits=110626, hitRatio=93.18%, , cachingAccesses=110745, cachingHits=110317, cachingHitsRatio=99.61%, evictions=12542, evicted=0, evictedPerRun=0.0 2022-03-06 14:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-06 14:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 14:32:10,682 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646576830681","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:58162","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:32:37,010 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646576857009","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:48662","multi.service_calls":0,"queuetimems":28525,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:32:37,010 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646576857010","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:44900","multi.service_calls":0,"queuetimems":26299,"class":"HRegionServer","multi.mutations":2} 2022-03-06 14:32:40,810 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646576860809","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300001,"client":"172.30.41.118:44914","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:32:58,102 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646576878102","responsesize":"6613","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:58162","multi.service_calls":0,"queuetimems":49618,"class":"HRegionServer","multi.mutations":4} 2022-03-06 14:33:09,042 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646576889042","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:44920","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:33:27,224 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646576907223","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:58164","multi.service_calls":0,"queuetimems":18181,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=119389, hits=111290, hitRatio=93.22%, , cachingAccesses=111409, cachingHits=110981, cachingHitsRatio=99.62%, evictions=12572, evicted=0, evictedPerRun=0.0 2022-03-06 14:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 14:37:08,486 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646576828484","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.155:58162","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":3} 2022-03-06 14:37:08,486 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646576828484","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 6 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.118:44906","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":6} 2022-03-06 14:37:10,683 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646577130682","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:44970","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:37:30,803 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646576850802","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 8 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:48662","multi.service_calls":0,"queuetimems":20120,"class":"HRegionServer","multi.mutations":8} 2022-03-06 14:37:37,318 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646577157010","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300308,"client":"172.31.0.219:48682","multi.service_calls":0,"queuetimems":26327,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:37:48,224 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646576568223","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 5 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:48634","multi.service_calls":0,"queuetimems":39740,"class":"HRegionServer","multi.mutations":5} 2022-03-06 14:37:58,686 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646576578636","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 8 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900050,"client":"172.31.0.219:48634","multi.service_calls":0,"queuetimems":50153,"class":"HRegionServer","multi.mutations":8} 2022-03-06 14:38:00,138 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646576880133","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 4 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600005,"client":"172.31.0.219:48662","multi.service_calls":0,"queuetimems":51649,"class":"HRegionServer","multi.mutations":4} 2022-03-06 14:38:04,478 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646576884477","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:58162","multi.service_calls":0,"queuetimems":53796,"class":"HRegionServer","multi.mutations":4} 2022-03-06 14:38:07,924 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 14:38:09,043 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646577189042","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:44970","multi.service_calls":0,"queuetimems":57839,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:38:09,043 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646576889042","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 5 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:58164","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":5} 2022-03-06 14:40:48,840 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2483ms No GCs detected 2022-03-06 14:40:52,726 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3386ms No GCs detected 2022-03-06 14:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=120021, hits=111922, hitRatio=93.25%, , cachingAccesses=112041, cachingHits=111613, cachingHitsRatio=99.62%, evictions=12602, evicted=0, evictedPerRun=0.0 2022-03-06 14:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 14:42:30,803 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646577450803","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:58196","multi.service_calls":0,"queuetimems":22316,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:42:37,011 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646577157010","responsesize":"11572","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 7 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:44970","multi.service_calls":0,"queuetimems":26328,"class":"HRegionServer","multi.mutations":7} 2022-03-06 14:42:40,811 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646577160810","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 6 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:58182","multi.service_calls":0,"queuetimems":30098,"class":"HRegionServer","multi.mutations":6} 2022-03-06 14:42:58,104 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646577178102","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 4 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.31.0.219:48682","multi.service_calls":0,"queuetimems":47419,"class":"HRegionServer","multi.mutations":4} 2022-03-06 14:42:58,688 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646577478686","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.31.0.219:48702","multi.service_calls":0,"queuetimems":50199,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:43:04,478 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646577484478","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:45052","multi.service_calls":0,"queuetimems":53794,"class":"HRegionServer","multi.mutations":2} 2022-03-06 14:43:09,043 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646577489043","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:48702","multi.service_calls":0,"queuetimems":58359,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:43:27,225 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646577207224","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:58182","multi.service_calls":0,"queuetimems":50213,"class":"HRegionServer","multi.mutations":2} 2022-03-06 14:44:10,310 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 14:46:29,460 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1393ms No GCs detected 2022-03-06 14:46:42,099 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 14:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=120688, hits=112589, hitRatio=93.29%, , cachingAccesses=112708, cachingHits=112280, cachingHitsRatio=99.62%, evictions=12632, evicted=0, evictedPerRun=0.0 2022-03-06 14:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 14:47:08,487 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646577428487","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:58196","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":4} 2022-03-06 14:47:08,487 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646577428486","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:45052","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 14:47:13,766 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 14:47:22,374 WARN [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 10170ms No GCs detected 2022-03-06 14:47:22,619 INFO [regionserver/cvp328:16201.Chore.1] hbase.ScheduledChore: Chore: CompactionChecker missed its start time 2022-03-06 14:47:22,619 INFO [regionserver/cvp328:16201.Chore.1] hbase.ScheduledChore: Chore: MemstoreFlusherChore missed its start time 2022-03-06 14:47:22,661 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 367ms. 2022-03-06 14:47:30,805 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646577750805","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:58214","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:47:35,321 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1520ms No GCs detected 2022-03-06 14:47:39,961 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646577757011","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":302949,"client":"172.30.41.118:45126","multi.service_calls":0,"queuetimems":6206,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:47:43,241 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 7419ms No GCs detected 2022-03-06 14:47:40,812 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646577760811","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:48718","multi.service_calls":0,"queuetimems":8983,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:47:49,064 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646577468224","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600840,"client":"172.30.41.155:58196","multi.service_calls":0,"queuetimems":37540,"class":"HRegionServer","multi.mutations":2} 2022-03-06 14:47:53,776 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 14:47:58,105 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646577778105","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:45126","multi.service_calls":0,"queuetimems":25165,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:47:58,688 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646577778688","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:48718","multi.service_calls":0,"queuetimems":24655,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:48:00,139 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646577480138","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:58196","multi.service_calls":0,"queuetimems":49454,"class":"HRegionServer","multi.mutations":5} 2022-03-06 14:48:04,479 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646577784478","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:45126","multi.service_calls":0,"queuetimems":30077,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:48:09,044 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646576889042","responsesize":"24794","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 15 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200001,"client":"172.30.41.118:44920","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":15} 2022-03-06 14:48:09,044 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646577789043","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:48718","multi.service_calls":0,"queuetimems":34671,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:48:09,044 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646577489044","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:58198","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-06 14:48:28,161 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2456ms No GCs detected 2022-03-06 14:48:28,338 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646577807225","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":301113,"client":"172.30.41.118:45126","multi.service_calls":0,"queuetimems":52824,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:48:36,701 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3407ms No GCs detected 2022-03-06 14:49:20,880 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3287ms No GCs detected 2022-03-06 14:50:28,673 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2204ms No GCs detected 2022-03-06 14:51:23,949 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1694ms No GCs detected 2022-03-06 14:51:25,940 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1491ms No GCs detected 2022-03-06 14:51:29,553 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1473ms No GCs detected 2022-03-06 14:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=121412, hits=113313, hitRatio=93.33%, , cachingAccesses=113432, cachingHits=113004, cachingHitsRatio=99.62%, evictions=12660, evicted=0, evictedPerRun=0.0 2022-03-06 14:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 14:52:58,106 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646578078106","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:58232","multi.service_calls":0,"queuetimems":48811,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:52:59,215 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646578078688","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300527,"client":"172.31.0.219:48738","multi.service_calls":0,"queuetimems":27881,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:53:00,142 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646578080139","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300003,"client":"172.30.41.155:58232","multi.service_calls":0,"queuetimems":29333,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:53:04,481 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646578084480","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:45194","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 14:53:09,045 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646578089044","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:45194","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 14:55:34,334 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 14:55:59,753 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1798ms No GCs detected 2022-03-06 14:56:18,311 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1054ms No GCs detected 2022-03-06 14:56:22,252 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1465ms No GCs detected 2022-03-06 14:56:27,860 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4108ms No GCs detected 2022-03-06 14:56:29,998 INFO [regionserver/cvp328:16201.Chore.1] hbase.ScheduledChore: Chore: CompactionChecker missed its start time 2022-03-06 14:56:29,998 INFO [regionserver/cvp328:16201.Chore.2] hbase.ScheduledChore: Chore: MemstoreFlusherChore missed its start time 2022-03-06 14:56:34,967 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 14:56:36,662 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1153ms No GCs detected 2022-03-06 14:56:44,455 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=122094, hits=113995, hitRatio=93.37%, , cachingAccesses=114114, cachingHits=113686, cachingHitsRatio=99.62%, evictions=12690, evicted=0, evictedPerRun=0.0 2022-03-06 14:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 14:57:08,489 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646578028489","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 6 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:45182","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":6} 2022-03-06 14:57:08,498 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646578028488","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 4 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600010,"client":"172.30.41.118:45182","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-06 14:57:09,941 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 14:57:10,686 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646577430683","responsesize":"23142","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 14 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200003,"client":"172.30.41.118:45052","multi.service_calls":0,"queuetimems":2196,"class":"HRegionServer","multi.mutations":14} 2022-03-06 14:57:26,705 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1025ms No GCs detected 2022-03-06 14:57:37,320 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646577457318","responsesize":"13230","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 8 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200002,"client":"172.31.0.219:48702","multi.service_calls":0,"queuetimems":28831,"class":"HRegionServer","multi.mutations":8} 2022-03-06 14:57:45,259 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646578065258","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:45182","multi.service_calls":0,"queuetimems":14452,"class":"HRegionServer","multi.mutations":4} 2022-03-06 14:57:59,217 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646578379217","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:45258","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:57:59,218 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646578379217","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300001,"client":"172.30.41.118:45258","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:58:02,214 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646578382214","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:58252","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:58:04,481 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646578384481","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:58252","multi.service_calls":0,"queuetimems":2267,"class":"HRegionServer","multi.mutations":1} 2022-03-06 14:58:09,045 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646578089044","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:48740","multi.service_calls":0,"queuetimems":4563,"class":"HRegionServer","multi.mutations":4} 2022-03-06 14:58:09,045 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646578089044","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 8 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:45194","multi.service_calls":0,"queuetimems":4564,"class":"HRegionServer","multi.mutations":8} 2022-03-06 14:58:09,045 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646578389045","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:45264","multi.service_calls":0,"queuetimems":9798,"class":"HRegionServer","multi.mutations":2} 2022-03-06 14:58:28,339 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646578108338","responsesize":"11572","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:48740","multi.service_calls":0,"queuetimems":23858,"class":"HRegionServer","multi.mutations":7} 2022-03-06 15:00:23,222 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1014ms No GCs detected 2022-03-06 15:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=122857, hits=114758, hitRatio=93.41%, , cachingAccesses=114877, cachingHits=114449, cachingHitsRatio=99.63%, evictions=12720, evicted=0, evictedPerRun=0.0 2022-03-06 15:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 15:02:09,668 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646578629668","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:45326","multi.service_calls":0,"queuetimems":1171,"class":"HRegionServer","multi.mutations":1} 2022-03-06 15:02:33,237 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646578050806","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":902431,"client":"172.31.0.219:48738","multi.service_calls":0,"queuetimems":22317,"class":"HRegionServer","multi.mutations":3} 2022-03-06 15:02:37,320 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646578657320","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:45326","multi.service_calls":0,"queuetimems":28823,"class":"HRegionServer","multi.mutations":1} 2022-03-06 15:02:45,260 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646578665259","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:48776","multi.service_calls":0,"queuetimems":36769,"class":"HRegionServer","multi.mutations":1} 2022-03-06 15:03:04,482 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646578684481","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:45334","multi.service_calls":0,"queuetimems":2266,"class":"HRegionServer","multi.mutations":1} 2022-03-06 15:03:09,046 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646578689046","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:58264","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 15:03:09,046 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646578689046","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:45334","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 15:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=123559, hits=115460, hitRatio=93.45%, , cachingAccesses=115579, cachingHits=115151, cachingHitsRatio=99.63%, evictions=12750, evicted=0, evictedPerRun=0.0 2022-03-06 15:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 15:07:08,491 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646578628490","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:58262","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 15:07:09,669 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646578929669","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:58278","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 15:07:33,238 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646578953237","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:45390","multi.service_calls":0,"queuetimems":23568,"class":"HRegionServer","multi.mutations":2} 2022-03-06 15:07:37,321 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646578957320","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:58278","multi.service_calls":0,"queuetimems":27651,"class":"HRegionServer","multi.mutations":2} 2022-03-06 15:07:45,240 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646578065238","responsesize":"16534","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 10 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200001,"client":"172.30.41.155:58232","multi.service_calls":0,"queuetimems":35943,"class":"HRegionServer","multi.mutations":10} 2022-03-06 15:07:49,067 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646578069064","responsesize":"21490","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 13 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200003,"client":"172.31.0.219:48738","multi.service_calls":0,"queuetimems":40575,"class":"HRegionServer","multi.mutations":13} 2022-03-06 15:07:59,219 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646578679218","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:48776","multi.service_calls":0,"queuetimems":48531,"class":"HRegionServer","multi.mutations":3} 2022-03-06 15:08:02,216 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646578682215","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:58264","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-06 15:08:04,482 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646578984482","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:48792","multi.service_calls":0,"queuetimems":54812,"class":"HRegionServer","multi.mutations":1} 2022-03-06 15:08:09,047 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646578989046","responsesize":"8265","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:48792","multi.service_calls":0,"queuetimems":59377,"class":"HRegionServer","multi.mutations":5} 2022-03-06 15:08:28,341 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646578708340","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 9 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:45334","multi.service_calls":0,"queuetimems":26125,"class":"HRegionServer","multi.mutations":9} 2022-03-06 15:08:50,089 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1100ms No GCs detected 2022-03-06 15:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=124362, hits=116263, hitRatio=93.49%, , cachingAccesses=116382, cachingHits=115954, cachingHitsRatio=99.63%, evictions=12780, evicted=0, evictedPerRun=0.0 2022-03-06 15:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 15:12:10,689 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646578630687","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 8 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.155:58262","multi.service_calls":0,"queuetimems":2197,"class":"HRegionServer","multi.mutations":8} 2022-03-06 15:12:59,317 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646578679218","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 7 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900099,"client":"172.31.0.219:48776","multi.service_calls":0,"queuetimems":50727,"class":"HRegionServer","multi.mutations":7} 2022-03-06 15:13:09,047 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646579289047","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:58310","multi.service_calls":0,"queuetimems":35808,"class":"HRegionServer","multi.mutations":1} 2022-03-06 15:13:28,373 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646579308373","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300000,"client":"172.30.41.118:45470","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-06 15:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=125072, hits=116973, hitRatio=93.52%, , cachingAccesses=117092, cachingHits=116664, cachingHitsRatio=99.63%, evictions=12810, evicted=0, evictedPerRun=0.0 2022-03-06 15:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 15:17:08,494 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646579228493","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:45462","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":3} 2022-03-06 15:17:10,690 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646579530690","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:45520","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 15:17:33,239 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646579253238","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:58310","multi.service_calls":0,"queuetimems":24745,"class":"HRegionServer","multi.mutations":2} 2022-03-06 15:17:45,261 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646578965260","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 8 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:58278","multi.service_calls":0,"queuetimems":35591,"class":"HRegionServer","multi.mutations":8} 2022-03-06 15:17:59,317 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646579279219","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600098,"client":"172.30.41.118:45462","multi.service_calls":0,"queuetimems":25980,"class":"HRegionServer","multi.mutations":3} 2022-03-06 15:18:02,217 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646579282216","responsesize":"11572","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 7 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:45462","multi.service_calls":0,"queuetimems":28977,"class":"HRegionServer","multi.mutations":7} 2022-03-06 15:18:04,484 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646579284483","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 8 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:45476","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":8} 2022-03-06 15:18:09,047 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646578689045","responsesize":"13230","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 8 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200002,"client":"172.31.0.219:48778","multi.service_calls":0,"queuetimems":6830,"class":"HRegionServer","multi.mutations":8} 2022-03-06 15:18:09,047 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646578989046","responsesize":"19835","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 12 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.30.41.118:45390","multi.service_calls":0,"queuetimems":59377,"class":"HRegionServer","multi.mutations":12} 2022-03-06 15:18:09,048 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646579589048","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:58346","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 15:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=125894, hits=117795, hitRatio=93.57%, , cachingAccesses=117914, cachingHits=117486, cachingHitsRatio=99.64%, evictions=12840, evicted=0, evictedPerRun=0.0 2022-03-06 15:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 15:22:08,495 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646579828495","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:45596","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 15:22:09,671 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646579229669","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 8 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.155:58310","multi.service_calls":0,"queuetimems":1176,"class":"HRegionServer","multi.mutations":8} 2022-03-06 15:22:33,240 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646579853239","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:45596","multi.service_calls":0,"queuetimems":22548,"class":"HRegionServer","multi.mutations":1} 2022-03-06 15:22:45,241 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646579265240","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 6 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:45462","multi.service_calls":0,"queuetimems":36747,"class":"HRegionServer","multi.mutations":6} 2022-03-06 15:22:45,261 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646579865261","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:48842","multi.service_calls":0,"queuetimems":34569,"class":"HRegionServer","multi.mutations":1} 2022-03-06 15:22:49,070 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646579269067","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 8 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900003,"client":"172.31.0.219:48808","multi.service_calls":0,"queuetimems":40574,"class":"HRegionServer","multi.mutations":8} 2022-03-06 15:22:59,318 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646579579317","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 6 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:45520","multi.service_calls":0,"queuetimems":48627,"class":"HRegionServer","multi.mutations":6} 2022-03-06 15:23:02,218 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646579882217","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:58370","multi.service_calls":0,"queuetimems":53722,"class":"HRegionServer","multi.mutations":1} 2022-03-06 15:23:09,048 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646579889048","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:45608","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 15:23:28,375 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646579608374","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:45540","multi.service_calls":0,"queuetimems":19326,"class":"HRegionServer","multi.mutations":3} 2022-03-06 15:26:44,758 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=126675, hits=118576, hitRatio=93.61%, , cachingAccesses=118695, cachingHits=118267, cachingHitsRatio=99.64%, evictions=12870, evicted=0, evictedPerRun=0.0 2022-03-06 15:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 15:27:08,497 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646580128496","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:45660","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 15:27:37,323 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646579257321","responsesize":"11578","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 7 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200002,"client":"172.31.0.219:48808","multi.service_calls":0,"queuetimems":28828,"class":"HRegionServer","multi.mutations":7} 2022-03-06 15:27:45,262 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646580165262","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:58404","multi.service_calls":0,"queuetimems":36477,"class":"HRegionServer","multi.mutations":1} 2022-03-06 15:27:59,318 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646579879317","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:58370","multi.service_calls":0,"queuetimems":50822,"class":"HRegionServer","multi.mutations":2} 2022-03-06 15:27:59,318 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646580179318","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:58404","multi.service_calls":0,"queuetimems":50533,"class":"HRegionServer","multi.mutations":1} 2022-03-06 15:28:02,218 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646580182218","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:58404","multi.service_calls":0,"queuetimems":53433,"class":"HRegionServer","multi.mutations":1} 2022-03-06 15:28:09,049 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646580189049","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:58404","multi.service_calls":0,"queuetimems":60264,"class":"HRegionServer","multi.mutations":1} 2022-03-06 15:28:09,049 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646579889048","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:58376","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-06 15:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=127408, hits=119309, hitRatio=93.64%, , cachingAccesses=119428, cachingHits=119000, cachingHitsRatio=99.64%, evictions=12900, evicted=0, evictedPerRun=0.0 2022-03-06 15:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-06 15:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 15:32:10,009 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646580130008","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:45660","multi.service_calls":0,"queuetimems":1511,"class":"HRegionServer","multi.mutations":4} 2022-03-06 15:32:45,264 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646580465263","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:58446","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 15:32:59,319 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646580479318","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:48878","multi.service_calls":0,"queuetimems":14055,"class":"HRegionServer","multi.mutations":2} 2022-03-06 15:32:59,319 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646580479319","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:48878","multi.service_calls":0,"queuetimems":14056,"class":"HRegionServer","multi.mutations":1} 2022-03-06 15:33:02,219 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646580482219","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.155:58444","multi.service_calls":0,"queuetimems":16926,"class":"HRegionServer","multi.mutations":2} 2022-03-06 15:33:09,049 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646579889048","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 8 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":900001,"client":"172.30.41.118:45608","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":8} 2022-03-06 15:33:09,049 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646580489049","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:48878","multi.service_calls":0,"queuetimems":9729,"class":"HRegionServer","multi.mutations":1} 2022-03-06 15:33:28,376 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646580208376","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:45676","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":2} 2022-03-06 15:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=128221, hits=120122, hitRatio=93.68%, , cachingAccesses=120241, cachingHits=119813, cachingHitsRatio=99.64%, evictions=12930, evicted=0, evictedPerRun=0.0 2022-03-06 15:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 15:37:08,499 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646580428498","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:58438","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 15:37:10,695 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646579830691","responsesize":"24794","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 15 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200004,"client":"172.30.41.118:45596","multi.service_calls":0,"queuetimems":2196,"class":"HRegionServer","multi.mutations":15} 2022-03-06 15:37:33,242 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646580153240","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:48860","multi.service_calls":0,"queuetimems":24743,"class":"HRegionServer","multi.mutations":4} 2022-03-06 15:37:45,243 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646580165241","responsesize":"14879","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 9 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.31.0.219:48860","multi.service_calls":0,"queuetimems":36744,"class":"HRegionServer","multi.mutations":9} 2022-03-06 15:37:45,265 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646580765265","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:58480","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-06 15:37:49,072 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646580169070","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.118:45660","multi.service_calls":0,"queuetimems":40573,"class":"HRegionServer","multi.mutations":4} 2022-03-06 15:37:59,321 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646580779320","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:45792","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 15:38:04,486 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646579884484","responsesize":"11578","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 7 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200002,"client":"172.31.0.219:48842","multi.service_calls":0,"queuetimems":55989,"class":"HRegionServer","multi.mutations":7} 2022-03-06 15:38:09,050 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646580789049","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:58480","multi.service_calls":0,"queuetimems":23784,"class":"HRegionServer","multi.mutations":2} 2022-03-06 15:38:35,358 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2437ms No GCs detected 2022-03-06 15:38:47,294 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 15:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=129014, hits=120915, hitRatio=93.72%, , cachingAccesses=121034, cachingHits=120606, cachingHitsRatio=99.65%, evictions=12960, evicted=0, evictedPerRun=0.0 2022-03-06 15:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 15:42:08,501 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646581028500","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300001,"client":"172.30.41.118:45846","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 15:42:10,011 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646580730010","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":600001,"client":"172.30.41.118:45782","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-06 15:42:33,242 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646581053242","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:45846","multi.service_calls":0,"queuetimems":22546,"class":"HRegionServer","multi.mutations":1} 2022-03-06 15:42:37,495 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646580457323","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900172,"client":"172.30.41.155:58438","multi.service_calls":0,"queuetimems":28825,"class":"HRegionServer","multi.mutations":4} 2022-03-06 15:42:45,244 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646581065243","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:45858","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 15:42:49,074 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646581069073","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:58512","multi.service_calls":0,"queuetimems":3829,"class":"HRegionServer","multi.mutations":1} 2022-03-06 15:43:02,220 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646580782219","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 5 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":600001,"client":"172.30.41.118:45792","multi.service_calls":0,"queuetimems":2899,"class":"HRegionServer","multi.mutations":5} 2022-03-06 15:43:04,487 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646581084486","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:48914","multi.service_calls":0,"queuetimems":19242,"class":"HRegionServer","multi.mutations":1} 2022-03-06 15:43:09,050 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646580789050","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 4 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:45792","multi.service_calls":0,"queuetimems":23785,"class":"HRegionServer","multi.mutations":4} 2022-03-06 15:43:09,050 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646581089050","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:48914","multi.service_calls":0,"queuetimems":23806,"class":"HRegionServer","multi.mutations":1} 2022-03-06 15:43:28,377 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646580808376","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:58480","multi.service_calls":0,"queuetimems":29056,"class":"HRegionServer","multi.mutations":5} 2022-03-06 15:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=129795, hits=121696, hitRatio=93.76%, , cachingAccesses=121815, cachingHits=121387, cachingHitsRatio=99.65%, evictions=12989, evicted=0, evictedPerRun=0.0 2022-03-06 15:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 15:47:08,502 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646581328502","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:45916","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 15:47:10,697 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646581030696","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 9 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:45846","multi.service_calls":0,"queuetimems":2195,"class":"HRegionServer","multi.mutations":9} 2022-03-06 15:47:45,266 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646581065265","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 6 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:58512","multi.service_calls":0,"queuetimems":21,"class":"HRegionServer","multi.mutations":6} 2022-03-06 15:47:59,320 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646580779319","responsesize":"14879","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 9 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:45792","multi.service_calls":0,"queuetimems":14054,"class":"HRegionServer","multi.mutations":9} 2022-03-06 15:48:09,051 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646580489049","responsesize":"23142","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 14 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.118:45742","multi.service_calls":0,"queuetimems":9730,"class":"HRegionServer","multi.mutations":14} 2022-03-06 15:48:09,051 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646581389051","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:45936","multi.service_calls":0,"queuetimems":4563,"class":"HRegionServer","multi.mutations":2} 2022-03-06 15:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=130574, hits=122475, hitRatio=93.80%, , cachingAccesses=122594, cachingHits=122166, cachingHitsRatio=99.65%, evictions=13019, evicted=0, evictedPerRun=0.0 2022-03-06 15:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 15:52:10,012 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646581330011","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 6 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":600001,"client":"172.30.41.118:45916","multi.service_calls":0,"queuetimems":1509,"class":"HRegionServer","multi.mutations":6} 2022-03-06 15:52:10,699 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646581630697","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.31.0.219:48952","multi.service_calls":0,"queuetimems":2193,"class":"HRegionServer","multi.mutations":1} 2022-03-06 15:52:33,243 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646581353243","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:58530","multi.service_calls":0,"queuetimems":24741,"class":"HRegionServer","multi.mutations":4} 2022-03-06 15:52:37,496 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646581357495","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:58530","multi.service_calls":0,"queuetimems":28993,"class":"HRegionServer","multi.mutations":3} 2022-03-06 15:52:45,245 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646581365244","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:48932","multi.service_calls":0,"queuetimems":36742,"class":"HRegionServer","multi.mutations":3} 2022-03-06 15:52:45,266 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646581665266","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:48952","multi.service_calls":0,"queuetimems":36762,"class":"HRegionServer","multi.mutations":1} 2022-03-06 15:52:49,075 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646581369074","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:48932","multi.service_calls":0,"queuetimems":40572,"class":"HRegionServer","multi.mutations":2} 2022-03-06 15:52:59,321 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646581679320","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:48952","multi.service_calls":0,"queuetimems":50816,"class":"HRegionServer","multi.mutations":1} 2022-03-06 15:53:04,489 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646581384488","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:58532","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 15:53:09,052 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646581689051","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:46002","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 15:53:09,052 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646581689051","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:58554","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 15:53:09,054 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646581389053","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 6 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:58532","multi.service_calls":0,"queuetimems":4565,"class":"HRegionServer","multi.mutations":6} 2022-03-06 15:53:28,904 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646581408378","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 5 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600526,"client":"172.31.0.219:48934","multi.service_calls":0,"queuetimems":23890,"class":"HRegionServer","multi.mutations":5} 2022-03-06 15:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=131320, hits=123221, hitRatio=93.83%, , cachingAccesses=123340, cachingHits=122912, cachingHitsRatio=99.65%, evictions=13049, evicted=0, evictedPerRun=0.0 2022-03-06 15:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 15:57:10,013 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646581930013","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:46046","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 15:57:10,700 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646581930700","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:58568","multi.service_calls":0,"queuetimems":687,"class":"HRegionServer","multi.mutations":1} 2022-03-06 15:57:37,496 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646581957496","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:58568","multi.service_calls":0,"queuetimems":27483,"class":"HRegionServer","multi.mutations":1} 2022-03-06 15:57:45,246 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646581965245","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:58568","multi.service_calls":0,"queuetimems":35232,"class":"HRegionServer","multi.mutations":1} 2022-03-06 15:57:45,267 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646581965266","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:58568","multi.service_calls":0,"queuetimems":35253,"class":"HRegionServer","multi.mutations":1} 2022-03-06 15:57:49,514 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646581969075","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300439,"client":"172.30.41.155:58568","multi.service_calls":0,"queuetimems":38375,"class":"HRegionServer","multi.mutations":1} 2022-03-06 15:57:59,324 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646581079321","responsesize":"23142","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 14 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.118:45858","multi.service_calls":0,"queuetimems":14077,"class":"HRegionServer","multi.mutations":14} 2022-03-06 15:57:59,410 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646581979323","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300087,"client":"172.30.41.118:46054","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-06 15:58:02,222 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646581382220","responsesize":"16531","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 10 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900002,"client":"172.30.41.118:45916","multi.service_calls":0,"queuetimems":28977,"class":"HRegionServer","multi.mutations":10} 2022-03-06 15:58:04,490 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646581984489","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:46054","multi.service_calls":0,"queuetimems":5167,"class":"HRegionServer","multi.mutations":1} 2022-03-06 15:58:09,053 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646581989052","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:46054","multi.service_calls":0,"queuetimems":9730,"class":"HRegionServer","multi.mutations":1} 2022-03-06 15:58:09,053 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646581989052","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:46054","multi.service_calls":0,"queuetimems":9730,"class":"HRegionServer","multi.mutations":1} 2022-03-06 15:58:09,054 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646581989054","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:46054","multi.service_calls":0,"queuetimems":9732,"class":"HRegionServer","multi.mutations":1} 2022-03-06 15:58:24,869 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 6446ms No GCs detected 2022-03-06 15:58:28,904 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646582008904","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:46054","multi.service_calls":0,"queuetimems":29582,"class":"HRegionServer","multi.mutations":1} 2022-03-06 15:58:55,364 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 15:59:33,184 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1159ms No GCs detected 2022-03-06 16:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=132070, hits=123971, hitRatio=93.87%, , cachingAccesses=124090, cachingHits=123662, cachingHitsRatio=99.66%, evictions=13079, evicted=0, evictedPerRun=0.0 2022-03-06 16:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 16:02:08,505 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646581628503","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.118:45990","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-06 16:02:10,015 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646582230014","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:58586","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-06 16:02:45,268 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646582265267","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:49024","multi.service_calls":0,"queuetimems":34565,"class":"HRegionServer","multi.mutations":3} 2022-03-06 16:02:59,446 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646582279445","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:46106","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 16:03:04,491 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646582284490","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:46112","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:03:09,054 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646582289054","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:46112","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:03:28,905 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646582308904","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:46112","multi.service_calls":0,"queuetimems":24413,"class":"HRegionServer","multi.mutations":2} 2022-03-06 16:05:55,533 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4559ms No GCs detected 2022-03-06 16:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=132856, hits=124757, hitRatio=93.90%, , cachingAccesses=124876, cachingHits=124448, cachingHitsRatio=99.66%, evictions=13109, evicted=0, evictedPerRun=0.0 2022-03-06 16:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 16:07:00,427 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 16:07:00,459 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 463ms. 2022-03-06 16:07:08,759 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646582528506","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300252,"client":"172.30.41.118:46158","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:07:10,015 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646582530015","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:46158","multi.service_calls":0,"queuetimems":1509,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:07:33,245 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646581953244","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 7 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:46046","multi.service_calls":0,"queuetimems":23231,"class":"HRegionServer","multi.mutations":7} 2022-03-06 16:07:37,668 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646582257496","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600172,"client":"172.31.0.219:49024","multi.service_calls":0,"queuetimems":27481,"class":"HRegionServer","multi.mutations":4} 2022-03-06 16:07:59,325 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646582279324","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:58586","multi.service_calls":0,"queuetimems":48623,"class":"HRegionServer","multi.mutations":2} 2022-03-06 16:08:04,492 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646582584492","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:58604","multi.service_calls":0,"queuetimems":19223,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:08:28,915 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646582608905","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300010,"client":"172.30.41.155:58604","multi.service_calls":0,"queuetimems":43636,"class":"HRegionServer","multi.mutations":2} 2022-03-06 16:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=133572, hits=125473, hitRatio=93.94%, , cachingAccesses=125592, cachingHits=125164, cachingHitsRatio=99.66%, evictions=13139, evicted=0, evictedPerRun=0.0 2022-03-06 16:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 16:12:08,766 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646582828765","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:49064","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:12:10,016 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646582830015","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:58622","multi.service_calls":0,"queuetimems":957,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:12:10,826 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646582230701","responsesize":"14879","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 9 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900125,"client":"172.31.0.219:49024","multi.service_calls":0,"queuetimems":686,"class":"HRegionServer","multi.mutations":9} 2022-03-06 16:12:33,247 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646582853246","responsesize":"9917","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 6 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:49064","multi.service_calls":0,"queuetimems":24481,"class":"HRegionServer","multi.mutations":6} 2022-03-06 16:12:37,669 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646582857668","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:58622","multi.service_calls":0,"queuetimems":28090,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:12:45,247 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646582265246","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:46096","multi.service_calls":0,"queuetimems":34545,"class":"HRegionServer","multi.mutations":3} 2022-03-06 16:12:59,327 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646582879325","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300002,"client":"172.30.41.118:46246","multi.service_calls":0,"queuetimems":49747,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:13:04,493 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646582884492","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:46246","multi.service_calls":0,"queuetimems":54913,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:13:09,056 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646582589054","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 6 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.155:58604","multi.service_calls":0,"queuetimems":23785,"class":"HRegionServer","multi.mutations":6} 2022-03-06 16:13:09,357 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646582289355","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.31.0.219:49030","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-06 16:13:28,917 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646582908916","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:46258","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=134303, hits=126204, hitRatio=93.97%, , cachingAccesses=126323, cachingHits=125895, cachingHitsRatio=99.66%, evictions=13169, evicted=0, evictedPerRun=0.0 2022-03-06 16:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 16:17:10,017 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646583130016","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:49080","multi.service_calls":0,"queuetimems":1249,"class":"HRegionServer","multi.mutations":2} 2022-03-06 16:17:45,249 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646583165248","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:46314","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-06 16:17:45,270 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646582565269","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:46170","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":3} 2022-03-06 16:17:59,328 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646583179327","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300001,"client":"172.30.41.118:46312","multi.service_calls":0,"queuetimems":21629,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:18:02,225 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":1,"starttimems":"1646582282222","responsesize":"12973","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 8 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200003,"client":"172.30.41.155:58586","multi.service_calls":0,"queuetimems":51521,"class":"HRegionServer","multi.mutations":7} 2022-03-06 16:18:09,056 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":1,"starttimems":"1646582289054","responsesize":"17929","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 11 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.155:58588","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":10} 2022-03-06 16:18:09,057 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646583189056","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:58640","multi.service_calls":0,"queuetimems":23808,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:18:09,357 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646583189357","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:58640","multi.service_calls":0,"queuetimems":4863,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:18:28,917 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646583208917","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:46314","multi.service_calls":0,"queuetimems":43669,"class":"HRegionServer","multi.mutations":3} 2022-03-06 16:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=135128, hits=127029, hitRatio=94.01%, , cachingAccesses=127148, cachingHits=126720, cachingHitsRatio=99.66%, evictions=13199, evicted=0, evictedPerRun=0.0 2022-03-06 16:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 16:22:08,768 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646583128767","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 8 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:46304","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":8} 2022-03-06 16:22:10,018 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646583430018","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:46382","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:22:10,829 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646583130827","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.118:46304","multi.service_calls":0,"queuetimems":2060,"class":"HRegionServer","multi.mutations":4} 2022-03-06 16:22:33,248 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646583153247","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:46304","multi.service_calls":0,"queuetimems":22420,"class":"HRegionServer","multi.mutations":3} 2022-03-06 16:22:37,670 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646583157669","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:58638","multi.service_calls":0,"queuetimems":26842,"class":"HRegionServer","multi.mutations":3} 2022-03-06 16:22:45,250 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646583465249","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:49098","multi.service_calls":0,"queuetimems":35231,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:22:49,887 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646582269883","responsesize":"26449","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 16 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1500004,"client":"172.30.41.118:46096","multi.service_calls":0,"queuetimems":39182,"class":"HRegionServer","multi.mutations":16} 2022-03-06 16:22:59,562 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646582579446","responsesize":"23142","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 14 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200116,"client":"172.30.41.118:46170","multi.service_calls":0,"queuetimems":14177,"class":"HRegionServer","multi.mutations":14} 2022-03-06 16:23:04,495 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646583184493","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 6 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.155:58640","multi.service_calls":0,"queuetimems":19245,"class":"HRegionServer","multi.mutations":6} 2022-03-06 16:23:09,089 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646583489088","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.31.0.219:49102","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 16:23:10,145 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646583490145","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300000,"client":"172.30.41.118:46394","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:23:28,918 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646583508918","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:49098","multi.service_calls":0,"queuetimems":77010,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=135910, hits=127811, hitRatio=94.04%, , cachingAccesses=127930, cachingHits=127502, cachingHitsRatio=99.67%, evictions=13229, evicted=0, evictedPerRun=0.0 2022-03-06 16:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 16:27:10,019 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646583730018","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:58672","multi.service_calls":0,"queuetimems":1249,"class":"HRegionServer","multi.mutations":2} 2022-03-06 16:27:10,831 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646583730830","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:46444","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:27:45,251 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646583765250","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:46460","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 16:27:49,890 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646583769888","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.155:58674","multi.service_calls":0,"queuetimems":4637,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:28:04,495 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646583784495","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:49132","multi.service_calls":0,"queuetimems":19244,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:28:09,057 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646583489056","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:58660","multi.service_calls":0,"queuetimems":23785,"class":"HRegionServer","multi.mutations":4} 2022-03-06 16:28:09,089 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646583789089","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:49132","multi.service_calls":0,"queuetimems":23838,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:28:10,146 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646583790145","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:49132","multi.service_calls":0,"queuetimems":24894,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=136626, hits=128527, hitRatio=94.07%, , cachingAccesses=128646, cachingHits=128218, cachingHitsRatio=99.67%, evictions=13259, evicted=0, evictedPerRun=0.0 2022-03-06 16:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-06 16:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 16:32:08,770 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646583728769","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:46444","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":5} 2022-03-06 16:32:37,672 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646583757670","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:49128","multi.service_calls":0,"queuetimems":28901,"class":"HRegionServer","multi.mutations":3} 2022-03-06 16:32:45,252 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646584065251","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:58694","multi.service_calls":0,"queuetimems":19930,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:32:59,329 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646583479328","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 7 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":900001,"client":"172.30.41.118:46382","multi.service_calls":0,"queuetimems":49310,"class":"HRegionServer","multi.mutations":7} 2022-03-06 16:33:02,227 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646583482225","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 6 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.31.0.219:49098","multi.service_calls":0,"queuetimems":52207,"class":"HRegionServer","multi.mutations":6} 2022-03-06 16:33:04,498 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646584084496","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.31.0.219:49168","multi.service_calls":0,"queuetimems":53664,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:33:09,058 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646584089057","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:46514","multi.service_calls":0,"queuetimems":34434,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:33:09,090 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646584089090","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:49168","multi.service_calls":0,"queuetimems":33843,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:33:10,147 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646584090146","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:49168","multi.service_calls":0,"queuetimems":34785,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:33:28,920 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646583808919","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:46468","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 16:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=137421, hits=129322, hitRatio=94.11%, , cachingAccesses=129441, cachingHits=129013, cachingHitsRatio=99.67%, evictions=13289, evicted=0, evictedPerRun=0.0 2022-03-06 16:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 16:37:08,772 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646584328771","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:46574","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 16:37:10,029 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646584030028","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:49168","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-06 16:37:10,834 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646584030832","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.118:46514","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 16:37:33,249 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646583753248","responsesize":"21487","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 13 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.30.41.118:46444","multi.service_calls":0,"queuetimems":22418,"class":"HRegionServer","multi.mutations":13} 2022-03-06 16:37:37,672 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646584357672","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:49186","multi.service_calls":0,"queuetimems":28900,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:37:45,253 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646584365253","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:46590","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:37:45,273 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646583465271","responsesize":"9926","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200002,"client":"172.30.41.155:58660","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":6} 2022-03-06 16:37:49,891 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646584069890","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:46514","multi.service_calls":0,"queuetimems":39058,"class":"HRegionServer","multi.mutations":3} 2022-03-06 16:37:59,331 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646584379330","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:58712","multi.service_calls":0,"queuetimems":14077,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:37:59,565 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646583779563","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 5 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.155:58674","multi.service_calls":0,"queuetimems":14312,"class":"HRegionServer","multi.mutations":5} 2022-03-06 16:38:02,228 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646584382227","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:58712","multi.service_calls":0,"queuetimems":16974,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:38:04,499 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646584384498","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:58712","multi.service_calls":0,"queuetimems":19245,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:38:09,058 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646584389058","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:58712","multi.service_calls":0,"queuetimems":23805,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:38:09,091 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646584389090","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:58712","multi.service_calls":0,"queuetimems":23837,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:38:10,147 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646584390147","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:58712","multi.service_calls":0,"queuetimems":24894,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=138109, hits=130010, hitRatio=94.14%, , cachingAccesses=130129, cachingHits=129701, cachingHitsRatio=99.67%, evictions=13319, evicted=0, evictedPerRun=0.0 2022-03-06 16:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 16:42:33,250 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646584653249","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:49206","multi.service_calls":0,"queuetimems":24476,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:42:37,673 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646584657672","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:58730","multi.service_calls":0,"queuetimems":28899,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:42:45,254 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646584665253","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:58730","multi.service_calls":0,"queuetimems":34418,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:42:55,441 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646584675439","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.155:58730","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:42:59,566 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646584679565","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:58730","multi.service_calls":0,"queuetimems":3376,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:42:59,567 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646584679565","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300002,"client":"172.30.41.118:46658","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:43:02,228 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646584682228","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:46658","multi.service_calls":0,"queuetimems":2299,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:43:04,499 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646584684499","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:58730","multi.service_calls":0,"queuetimems":2999,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:43:09,059 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646584689058","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.31.0.219:49166","multi.service_calls":0,"queuetimems":11679,"class":"HRegionServer","multi.mutations":2} 2022-03-06 16:43:09,091 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646584689091","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:58730","multi.service_calls":0,"queuetimems":7414,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:43:10,148 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646584690147","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300001,"client":"172.30.41.118:46660","multi.service_calls":0,"queuetimems":7891,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:43:28,921 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646584408920","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:46590","multi.service_calls":0,"queuetimems":43667,"class":"HRegionServer","multi.mutations":4} 2022-03-06 16:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=138865, hits=130766, hitRatio=94.17%, , cachingAccesses=130885, cachingHits=130457, cachingHitsRatio=99.67%, evictions=13349, evicted=0, evictedPerRun=0.0 2022-03-06 16:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 16:47:08,774 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646584628773","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:46656","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 16:47:10,836 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646584630835","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:58730","multi.service_calls":0,"queuetimems":2062,"class":"HRegionServer","multi.mutations":4} 2022-03-06 16:47:33,252 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646584953251","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:58748","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:47:37,674 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646584957673","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:46716","multi.service_calls":0,"queuetimems":4422,"class":"HRegionServer","multi.mutations":2} 2022-03-06 16:47:45,255 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646584965254","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:58748","multi.service_calls":0,"queuetimems":12003,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:47:55,442 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646584975441","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:58748","multi.service_calls":0,"queuetimems":22190,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:47:59,567 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646584979566","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:58748","multi.service_calls":0,"queuetimems":26315,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:47:59,567 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646584979567","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300000,"client":"172.30.41.118:46716","multi.service_calls":0,"queuetimems":26316,"class":"HRegionServer","multi.mutations":2} 2022-03-06 16:48:02,229 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646584982228","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:58748","multi.service_calls":0,"queuetimems":28977,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:48:04,500 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646584984499","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:49244","multi.service_calls":0,"queuetimems":31248,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:48:09,059 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646584989059","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:58748","multi.service_calls":0,"queuetimems":35808,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:48:09,092 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646584989091","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:46716","multi.service_calls":0,"queuetimems":31417,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:48:28,923 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646585008922","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:58750","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=139538, hits=131439, hitRatio=94.20%, , cachingAccesses=131558, cachingHits=131130, cachingHitsRatio=99.67%, evictions=13379, evicted=0, evictedPerRun=0.0 2022-03-06 16:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 16:52:08,776 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646585228775","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:46772","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 16:52:10,031 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646584630029","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 7 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.118:46656","multi.service_calls":0,"queuetimems":1256,"class":"HRegionServer","multi.mutations":7} 2022-03-06 16:52:33,252 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646585253252","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:58766","multi.service_calls":0,"queuetimems":24477,"class":"HRegionServer","multi.mutations":2} 2022-03-06 16:52:37,675 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646585257674","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:49274","multi.service_calls":0,"queuetimems":28899,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:52:45,255 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646585265255","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:58766","multi.service_calls":0,"queuetimems":36480,"class":"HRegionServer","multi.mutations":2} 2022-03-06 16:52:45,274 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646584665273","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:58730","multi.service_calls":0,"queuetimems":34438,"class":"HRegionServer","multi.mutations":6} 2022-03-06 16:52:55,442 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646585275442","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:49274","multi.service_calls":0,"queuetimems":46667,"class":"HRegionServer","multi.mutations":3} 2022-03-06 16:52:59,568 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646585279567","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:58766","multi.service_calls":0,"queuetimems":48730,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:52:59,568 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646585279567","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300001,"client":"172.30.41.118:46772","multi.service_calls":0,"queuetimems":48730,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:53:04,501 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646585284500","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:49274","multi.service_calls":0,"queuetimems":53663,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:53:09,061 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646585289060","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300000,"client":"172.30.41.118:46780","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:53:09,092 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646585289092","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:46780","multi.service_calls":0,"queuetimems":32,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:53:28,924 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646585308924","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:58768","multi.service_calls":0,"queuetimems":3365,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=140351, hits=132252, hitRatio=94.23%, , cachingAccesses=132371, cachingHits=131943, cachingHitsRatio=99.68%, evictions=13409, evicted=0, evictedPerRun=0.0 2022-03-06 16:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 16:57:10,031 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646585530031","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:58792","multi.service_calls":0,"queuetimems":1254,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:57:10,864 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646585230836","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600028,"client":"172.30.41.118:46772","multi.service_calls":0,"queuetimems":2061,"class":"HRegionServer","multi.mutations":3} 2022-03-06 16:57:45,275 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646585565274","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:46822","multi.service_calls":0,"queuetimems":12021,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:57:59,568 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646585579568","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:46822","multi.service_calls":0,"queuetimems":26315,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:58:04,502 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646585584501","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:46832","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 16:58:09,061 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646585589061","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:49300","multi.service_calls":0,"queuetimems":35807,"class":"HRegionServer","multi.mutations":2} 2022-03-06 16:58:10,149 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646584990148","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 8 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:46716","multi.service_calls":0,"queuetimems":32474,"class":"HRegionServer","multi.mutations":8} 2022-03-06 16:58:28,925 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646585608924","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:58798","multi.service_calls":0,"queuetimems":24423,"class":"HRegionServer","multi.mutations":2} 2022-03-06 17:01:07,418 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 17:01:07,421 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 462ms. 2022-03-06 17:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=141131, hits=133032, hitRatio=94.26%, , cachingAccesses=133151, cachingHits=132723, cachingHitsRatio=99.68%, evictions=13439, evicted=0, evictedPerRun=0.0 2022-03-06 17:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 17:02:08,778 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646585528777","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:46822","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 17:02:09,369 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 17:02:09,535 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 474ms. 2022-03-06 17:02:10,033 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646585830032","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:58830","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:02:33,254 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646585553253","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:58792","multi.service_calls":0,"queuetimems":24476,"class":"HRegionServer","multi.mutations":2} 2022-03-06 17:02:55,444 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646585575443","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:58792","multi.service_calls":0,"queuetimems":22190,"class":"HRegionServer","multi.mutations":2} 2022-03-06 17:02:59,573 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646585879568","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300005,"client":"172.30.41.155:58830","multi.service_calls":0,"queuetimems":49535,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:03:02,231 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646585282229","responsesize":"21487","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 13 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.30.41.118:46772","multi.service_calls":0,"queuetimems":51392,"class":"HRegionServer","multi.mutations":13} 2022-03-06 17:03:04,502 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646585884502","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:58830","multi.service_calls":0,"queuetimems":54469,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:03:09,062 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646585889061","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:58830","multi.service_calls":0,"queuetimems":59028,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:03:09,093 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646585589092","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 9 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:49300","multi.service_calls":0,"queuetimems":35838,"class":"HRegionServer","multi.mutations":9} 2022-03-06 17:03:10,150 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646585890149","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:58830","multi.service_calls":0,"queuetimems":59284,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=142003, hits=133904, hitRatio=94.30%, , cachingAccesses=134023, cachingHits=133595, cachingHitsRatio=99.68%, evictions=13469, evicted=0, evictedPerRun=0.0 2022-03-06 17:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 17:07:10,034 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646586130033","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:58860","multi.service_calls":0,"queuetimems":495,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:07:10,867 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646585830865","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 5 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.118:46872","multi.service_calls":0,"queuetimems":833,"class":"HRegionServer","multi.mutations":5} 2022-03-06 17:07:37,676 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646585557675","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 6 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:49300","multi.service_calls":0,"queuetimems":28898,"class":"HRegionServer","multi.mutations":6} 2022-03-06 17:07:45,276 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646585865275","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 9 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:46872","multi.service_calls":0,"queuetimems":35242,"class":"HRegionServer","multi.mutations":9} 2022-03-06 17:07:55,445 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646586175445","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:58860","multi.service_calls":0,"queuetimems":45907,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:07:59,574 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646585579568","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 7 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900006,"client":"172.30.41.155:58792","multi.service_calls":0,"queuetimems":26315,"class":"HRegionServer","multi.mutations":7} 2022-03-06 17:07:59,574 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646586179573","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:58860","multi.service_calls":0,"queuetimems":50034,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:08:02,231 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646586182231","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:58860","multi.service_calls":0,"queuetimems":52692,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:08:04,503 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646586184503","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:46928","multi.service_calls":0,"queuetimems":31248,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:08:09,094 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646586189093","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:49336","multi.service_calls":0,"queuetimems":30,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:08:28,927 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646585908926","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:46882","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 17:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=142882, hits=134783, hitRatio=94.33%, , cachingAccesses=134902, cachingHits=134474, cachingHitsRatio=99.68%, evictions=13499, evicted=0, evictedPerRun=0.0 2022-03-06 17:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 17:12:08,780 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646586128779","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 8 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:46928","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":8} 2022-03-06 17:12:10,035 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646586430035","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:58894","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:12:10,868 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646586430867","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:47000","multi.service_calls":0,"queuetimems":832,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:12:33,255 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646586153254","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 4 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:46928","multi.service_calls":0,"queuetimems":24475,"class":"HRegionServer","multi.mutations":4} 2022-03-06 17:12:45,257 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646585565255","responsesize":"24794","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 15 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.118:46822","multi.service_calls":0,"queuetimems":36478,"class":"HRegionServer","multi.mutations":15} 2022-03-06 17:12:45,276 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646586465276","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:58894","multi.service_calls":0,"queuetimems":35241,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:13:09,094 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646586489094","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:47000","multi.service_calls":0,"queuetimems":58226,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:13:10,151 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646586190150","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 4 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:49336","multi.service_calls":0,"queuetimems":1087,"class":"HRegionServer","multi.mutations":4} 2022-03-06 17:13:10,246 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646586490246","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:49354","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:13:28,928 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646586508928","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:47014","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 17:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=143646, hits=135547, hitRatio=94.36%, , cachingAccesses=135666, cachingHits=135238, cachingHitsRatio=99.68%, evictions=13529, evicted=0, evictedPerRun=0.0 2022-03-06 17:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 17:17:08,782 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646586728781","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:47062","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:17:10,036 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646586730035","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:49370","multi.service_calls":0,"queuetimems":1253,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:17:33,256 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646586753255","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:58928","multi.service_calls":0,"queuetimems":24473,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:17:37,677 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646586457676","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:47000","multi.service_calls":0,"queuetimems":27641,"class":"HRegionServer","multi.mutations":4} 2022-03-06 17:17:59,575 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646586479574","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:49352","multi.service_calls":0,"queuetimems":49539,"class":"HRegionServer","multi.mutations":4} 2022-03-06 17:18:09,063 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646586189062","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 7 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:49336","multi.service_calls":0,"queuetimems":4558,"class":"HRegionServer","multi.mutations":7} 2022-03-06 17:18:09,095 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646586789094","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300001,"client":"172.30.41.118:47062","multi.service_calls":0,"queuetimems":42921,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:18:10,152 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646586790152","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:58932","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:18:10,247 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646586790246","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:47062","multi.service_calls":0,"queuetimems":59377,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:18:28,929 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646586808929","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:58932","multi.service_calls":0,"queuetimems":18777,"class":"HRegionServer","multi.mutations":3} 2022-03-06 17:19:11,178 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 17:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=144411, hits=136312, hitRatio=94.39%, , cachingAccesses=136431, cachingHits=136003, cachingHitsRatio=99.69%, evictions=13559, evicted=0, evictedPerRun=0.0 2022-03-06 17:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 17:22:23,886 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1465ms No GCs detected 2022-03-06 17:22:37,678 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646587057677","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.155:58964","multi.service_calls":0,"queuetimems":28864,"class":"HRegionServer","multi.mutations":2} 2022-03-06 17:22:45,258 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646586765257","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:47062","multi.service_calls":0,"queuetimems":34388,"class":"HRegionServer","multi.mutations":4} 2022-03-06 17:22:45,278 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646586765276","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 5 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.118:47062","multi.service_calls":0,"queuetimems":34407,"class":"HRegionServer","multi.mutations":5} 2022-03-06 17:22:52,183 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1127ms No GCs detected 2022-03-06 17:22:59,576 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646586479574","responsesize":"19835","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 12 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900002,"client":"172.30.41.118:47000","multi.service_calls":0,"queuetimems":48706,"class":"HRegionServer","multi.mutations":12} 2022-03-06 17:23:02,233 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646586482231","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:49352","multi.service_calls":0,"queuetimems":51363,"class":"HRegionServer","multi.mutations":6} 2022-03-06 17:23:09,065 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646587089064","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:49390","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:23:10,248 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646587090247","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:47124","multi.service_calls":0,"queuetimems":36990,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:23:28,930 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646587108930","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:47124","multi.service_calls":0,"queuetimems":55673,"class":"HRegionServer","multi.mutations":3} 2022-03-06 17:25:42,297 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 17:26:14,103 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 17:26:17,862 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1231ms No GCs detected 2022-03-06 17:26:19,333 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 779ms. 2022-03-06 17:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=145212, hits=137113, hitRatio=94.42%, , cachingAccesses=137232, cachingHits=136804, cachingHitsRatio=99.69%, evictions=13589, evicted=0, evictedPerRun=0.0 2022-03-06 17:26:48,080 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 17:26:50,212 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 17:27:08,784 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646587028783","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:47124","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-06 17:27:10,910 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646586730868","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900042,"client":"172.30.41.118:47062","multi.service_calls":0,"queuetimems":2087,"class":"HRegionServer","multi.mutations":7} 2022-03-06 17:27:55,449 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646586475445","responsesize":"16534","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 10 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200004,"client":"172.30.41.155:58894","multi.service_calls":0,"queuetimems":45410,"class":"HRegionServer","multi.mutations":10} 2022-03-06 17:27:59,577 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646587079575","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 9 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.31.0.219:49388","multi.service_calls":0,"queuetimems":26318,"class":"HRegionServer","multi.mutations":9} 2022-03-06 17:28:02,233 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646587382233","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:49418","multi.service_calls":0,"queuetimems":24554,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:28:09,065 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646587389065","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:49418","multi.service_calls":0,"queuetimems":31386,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:28:09,096 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646587089095","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 4 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:49390","multi.service_calls":0,"queuetimems":30,"class":"HRegionServer","multi.mutations":4} 2022-03-06 17:28:10,248 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646587390248","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:49418","multi.service_calls":0,"queuetimems":32569,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:28:10,248 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646587090248","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:58966","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 17:28:28,931 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646587408930","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:47184","multi.service_calls":0,"queuetimems":43671,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:28:47,945 INFO [regionserver/cvp328:16201.Chore.3] hbase.ScheduledChore: Chore: MemstoreFlusherChore missed its start time 2022-03-06 17:28:47,945 INFO [regionserver/cvp328:16201.Chore.2] hbase.ScheduledChore: Chore: CompactionChecker missed its start time 2022-03-06 17:28:47,947 WARN [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 10305ms No GCs detected 2022-03-06 17:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=145989, hits=137890, hitRatio=94.45%, , cachingAccesses=138009, cachingHits=137581, cachingHitsRatio=99.69%, evictions=13619, evicted=0, evictedPerRun=0.0 2022-03-06 17:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-06 17:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 17:32:10,906 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646587030036","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 7 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900870,"client":"172.31.0.219:49388","multi.service_calls":0,"queuetimems":1253,"class":"HRegionServer","multi.mutations":7} 2022-03-06 17:32:37,680 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646587357679","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:47184","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-06 17:32:59,577 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646587379576","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:59008","multi.service_calls":0,"queuetimems":21897,"class":"HRegionServer","multi.mutations":4} 2022-03-06 17:33:09,067 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646587689067","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:47244","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:33:09,096 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646587689096","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:47244","multi.service_calls":0,"queuetimems":29,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:33:10,249 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646587690249","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:47244","multi.service_calls":0,"queuetimems":1182,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:33:10,249 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646587690249","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:47244","multi.service_calls":0,"queuetimems":1182,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:33:43,598 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1191ms No GCs detected 2022-03-06 17:35:14,630 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1243ms No GCs detected 2022-03-06 17:35:26,788 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2971ms No GCs detected 2022-03-06 17:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=146808, hits=138709, hitRatio=94.48%, , cachingAccesses=138828, cachingHits=138400, cachingHitsRatio=99.69%, evictions=13649, evicted=0, evictedPerRun=0.0 2022-03-06 17:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 17:37:08,786 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646587628785","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:47232","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 17:37:10,908 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646587930907","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:59042","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 17:37:10,913 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646587630911","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.155:59026","multi.service_calls":0,"queuetimems":2126,"class":"HRegionServer","multi.mutations":5} 2022-03-06 17:37:33,258 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646587053256","responsesize":"19838","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 12 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.118:47124","multi.service_calls":0,"queuetimems":24473,"class":"HRegionServer","multi.mutations":12} 2022-03-06 17:37:40,633 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1460ms No GCs detected 2022-03-06 17:37:45,503 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646587365278","responsesize":"14879","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 9 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900225,"client":"172.30.41.118:47184","multi.service_calls":0,"queuetimems":7599,"class":"HRegionServer","multi.mutations":9} 2022-03-06 17:37:55,450 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646587675449","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:59026","multi.service_calls":0,"queuetimems":46664,"class":"HRegionServer","multi.mutations":2} 2022-03-06 17:37:59,578 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646587979578","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:47300","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:37:59,578 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646587679577","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:49456","multi.service_calls":0,"queuetimems":50785,"class":"HRegionServer","multi.mutations":4} 2022-03-06 17:38:02,234 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646587682233","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 8 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:49456","multi.service_calls":0,"queuetimems":51321,"class":"HRegionServer","multi.mutations":8} 2022-03-06 17:38:09,068 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646587989068","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:47300","multi.service_calls":0,"queuetimems":9490,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:38:09,097 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646587989096","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:47300","multi.service_calls":0,"queuetimems":9518,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:38:10,250 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646587990249","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:47300","multi.service_calls":0,"queuetimems":10671,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:38:10,250 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646587990249","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:47300","multi.service_calls":0,"queuetimems":10671,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:38:30,494 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1177ms No GCs detected 2022-03-06 17:38:33,139 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1145ms No GCs detected 2022-03-06 17:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=147572, hits=139473, hitRatio=94.51%, , cachingAccesses=139592, cachingHits=139164, cachingHitsRatio=99.69%, evictions=13679, evicted=0, evictedPerRun=0.0 2022-03-06 17:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 17:42:08,788 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646588228787","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:47360","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:42:10,914 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646588230913","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.30.41.118:47354","multi.service_calls":0,"queuetimems":2096,"class":"HRegionServer","multi.mutations":2} 2022-03-06 17:42:33,259 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646588253258","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:59062","multi.service_calls":0,"queuetimems":23833,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:42:37,681 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646587957680","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:47286","multi.service_calls":0,"queuetimems":26773,"class":"HRegionServer","multi.mutations":3} 2022-03-06 17:42:45,261 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646587365259","responsesize":"9926","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200001,"client":"172.30.41.155:59008","multi.service_calls":0,"queuetimems":7580,"class":"HRegionServer","multi.mutations":6} 2022-03-06 17:42:45,608 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646588265608","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:47360","multi.service_calls":0,"queuetimems":34694,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:42:59,579 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646588279578","responsesize":"11569","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:49514","multi.service_calls":0,"queuetimems":50790,"class":"HRegionServer","multi.mutations":7} 2022-03-06 17:43:02,235 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646588282234","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:49514","multi.service_calls":0,"queuetimems":51320,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:43:09,128 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646588289128","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:47364","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 17:43:10,478 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646588290250","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300228,"client":"172.30.41.155:59064","multi.service_calls":0,"queuetimems":745,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:43:10,484 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646588290250","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300234,"client":"172.30.41.118:47362","multi.service_calls":0,"queuetimems":883,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:43:29,063 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646587708931","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 8 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900132,"client":"172.31.0.219:49460","multi.service_calls":0,"queuetimems":19864,"class":"HRegionServer","multi.mutations":8} 2022-03-06 17:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=148333, hits=140234, hitRatio=94.54%, , cachingAccesses=140353, cachingHits=139925, cachingHitsRatio=99.70%, evictions=13709, evicted=0, evictedPerRun=0.0 2022-03-06 17:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 17:47:08,789 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646588528789","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:47412","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:47:10,909 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646588230908","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 5 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:47360","multi.service_calls":0,"queuetimems":2121,"class":"HRegionServer","multi.mutations":5} 2022-03-06 17:47:37,681 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646588557681","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:59080","multi.service_calls":0,"queuetimems":28892,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:47:45,262 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646588565261","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:47426","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:47:55,452 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646588275450","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.31.0.219:49514","multi.service_calls":0,"queuetimems":46663,"class":"HRegionServer","multi.mutations":4} 2022-03-06 17:48:09,129 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646588589128","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.30.41.118:47424","multi.service_calls":0,"queuetimems":23837,"class":"HRegionServer","multi.mutations":2} 2022-03-06 17:48:10,478 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646588590478","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:49532","multi.service_calls":0,"queuetimems":25216,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:48:10,484 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646588590484","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300000,"client":"172.30.41.118:47426","multi.service_calls":0,"queuetimems":10904,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=149067, hits=140968, hitRatio=94.57%, , cachingAccesses=141087, cachingHits=140659, cachingHitsRatio=99.70%, evictions=13739, evicted=0, evictedPerRun=0.0 2022-03-06 17:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 17:52:08,791 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646588828790","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:47486","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:52:10,910 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646588830909","responsesize":"9917","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:47486","multi.service_calls":0,"queuetimems":2119,"class":"HRegionServer","multi.mutations":6} 2022-03-06 17:52:10,916 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646588530914","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 5 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.118:47412","multi.service_calls":0,"queuetimems":2125,"class":"HRegionServer","multi.mutations":5} 2022-03-06 17:52:37,682 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646588857681","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:59098","multi.service_calls":0,"queuetimems":28891,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:52:45,263 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646588865263","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:47500","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:52:45,609 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646588565608","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:59082","multi.service_calls":0,"queuetimems":347,"class":"HRegionServer","multi.mutations":4} 2022-03-06 17:52:55,453 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646588875453","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:59100","multi.service_calls":0,"queuetimems":10190,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:52:59,582 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646588279578","responsesize":"19835","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 12 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900004,"client":"172.30.41.118:47360","multi.service_calls":0,"queuetimems":48664,"class":"HRegionServer","multi.mutations":12} 2022-03-06 17:53:09,070 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646588289068","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 5 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.31.0.219:49514","multi.service_calls":0,"queuetimems":58154,"class":"HRegionServer","multi.mutations":5} 2022-03-06 17:53:09,129 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646588889129","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:59100","multi.service_calls":0,"queuetimems":23866,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:53:29,064 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646588609063","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 5 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:49532","multi.service_calls":0,"queuetimems":43801,"class":"HRegionServer","multi.mutations":5} 2022-03-06 17:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=149822, hits=141723, hitRatio=94.59%, , cachingAccesses=141842, cachingHits=141414, cachingHitsRatio=99.70%, evictions=13769, evicted=0, evictedPerRun=0.0 2022-03-06 17:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 17:57:08,792 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646589128792","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:47566","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:57:10,911 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646589130910","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:59120","multi.service_calls":0,"queuetimems":2118,"class":"HRegionServer","multi.mutations":2} 2022-03-06 17:57:33,260 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646588553259","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 5 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:49530","multi.service_calls":0,"queuetimems":24470,"class":"HRegionServer","multi.mutations":5} 2022-03-06 17:57:45,264 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646589165264","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:59120","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:57:45,610 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646589165609","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:59120","multi.service_calls":0,"queuetimems":345,"class":"HRegionServer","multi.mutations":2} 2022-03-06 17:57:59,581 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646588579580","responsesize":"18183","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 11 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.30.41.118:47426","multi.service_calls":0,"queuetimems":14319,"class":"HRegionServer","multi.mutations":11} 2022-03-06 17:57:59,583 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646589179582","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:49574","multi.service_calls":0,"queuetimems":14317,"class":"HRegionServer","multi.mutations":2} 2022-03-06 17:58:02,236 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646588582235","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 5 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:59082","multi.service_calls":0,"queuetimems":16973,"class":"HRegionServer","multi.mutations":5} 2022-03-06 17:58:09,070 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646589189070","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:59120","multi.service_calls":0,"queuetimems":13616,"class":"HRegionServer","multi.mutations":1} 2022-03-06 17:58:10,480 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646588890479","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 9 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:47500","multi.service_calls":0,"queuetimems":25216,"class":"HRegionServer","multi.mutations":9} 2022-03-06 17:58:10,485 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646588890484","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 5 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:49552","multi.service_calls":0,"queuetimems":25221,"class":"HRegionServer","multi.mutations":5} 2022-03-06 17:58:29,066 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646589209065","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:59124","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=150520, hits=142421, hitRatio=94.62%, , cachingAccesses=142540, cachingHits=142112, cachingHitsRatio=99.70%, evictions=13799, evicted=0, evictedPerRun=0.0 2022-03-06 18:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 18:02:10,911 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646589430911","responsesize":"6613","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:47610","multi.service_calls":0,"queuetimems":2118,"class":"HRegionServer","multi.mutations":4} 2022-03-06 18:02:33,261 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646589453260","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:47610","multi.service_calls":0,"queuetimems":24467,"class":"HRegionServer","multi.mutations":3} 2022-03-06 18:02:37,683 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646589157682","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 4 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:49572","multi.service_calls":0,"queuetimems":28890,"class":"HRegionServer","multi.mutations":4} 2022-03-06 18:02:45,265 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646589465265","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:59148","multi.service_calls":0,"queuetimems":36472,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:02:45,610 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646589465610","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:49592","multi.service_calls":0,"queuetimems":36816,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:02:55,456 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646589175454","responsesize":"11572","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 7 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.118:47572","multi.service_calls":0,"queuetimems":10190,"class":"HRegionServer","multi.mutations":7} 2022-03-06 18:02:59,583 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646589479581","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.31.0.219:49592","multi.service_calls":0,"queuetimems":50787,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:02:59,583 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646589479583","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:49592","multi.service_calls":0,"queuetimems":50789,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:03:02,237 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646589482236","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:49592","multi.service_calls":0,"queuetimems":53442,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:03:09,071 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646589489070","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:49592","multi.service_calls":0,"queuetimems":60276,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:03:10,480 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646589490480","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:49592","multi.service_calls":0,"queuetimems":61686,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:03:10,486 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646589490485","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:49592","multi.service_calls":0,"queuetimems":61691,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:03:29,066 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646589509066","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:59148","multi.service_calls":0,"queuetimems":55805,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=151202, hits=143103, hitRatio=94.64%, , cachingAccesses=143222, cachingHits=142794, cachingHitsRatio=99.70%, evictions=13829, evicted=0, evictedPerRun=0.0 2022-03-06 18:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 18:07:08,794 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646589428793","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:59148","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 18:07:10,912 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646589730912","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.31.0.219:49496","multi.service_calls":0,"queuetimems":12608,"class":"HRegionServer","multi.mutations":2} 2022-03-06 18:07:10,919 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646589130917","responsesize":"18183","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 11 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900002,"client":"172.30.41.118:47566","multi.service_calls":0,"queuetimems":2125,"class":"HRegionServer","multi.mutations":11} 2022-03-06 18:07:45,266 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646589765265","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:47660","multi.service_calls":0,"queuetimems":34021,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:07:45,611 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646589765610","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:47660","multi.service_calls":0,"queuetimems":32243,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:07:59,585 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646589779583","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.118:47660","multi.service_calls":0,"queuetimems":46996,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:07:59,585 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646589779583","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.31.0.219:49608","multi.service_calls":0,"queuetimems":45860,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:08:02,237 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646589782237","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:49608","multi.service_calls":0,"queuetimems":47133,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:08:09,071 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646589789071","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:47660","multi.service_calls":0,"queuetimems":54194,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:08:09,131 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646589189130","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 6 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:47572","multi.service_calls":0,"queuetimems":13676,"class":"HRegionServer","multi.mutations":6} 2022-03-06 18:08:10,481 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646589790480","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:49608","multi.service_calls":0,"queuetimems":53640,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:08:10,486 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646589790486","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:47660","multi.service_calls":0,"queuetimems":37224,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:08:29,067 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646589809066","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:59186","multi.service_calls":0,"queuetimems":55804,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=151969, hits=143870, hitRatio=94.67%, , cachingAccesses=143989, cachingHits=143561, cachingHitsRatio=99.70%, evictions=13859, evicted=0, evictedPerRun=0.0 2022-03-06 18:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 18:12:10,913 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646590030912","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:59200","multi.service_calls":0,"queuetimems":2117,"class":"HRegionServer","multi.mutations":2} 2022-03-06 18:12:10,920 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646590030920","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:49644","multi.service_calls":0,"queuetimems":2124,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:12:45,266 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646590065266","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:47712","multi.service_calls":0,"queuetimems":34346,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:12:55,458 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646589775456","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 6 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.31.0.219:49608","multi.service_calls":0,"queuetimems":44543,"class":"HRegionServer","multi.mutations":6} 2022-03-06 18:12:59,630 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646590079630","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300000,"client":"172.30.41.118:47722","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:13:02,238 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646590082237","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:47712","multi.service_calls":0,"queuetimems":51317,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:13:09,103 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646590089103","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.31.0.219:49652","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 18:13:10,479 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646590090479","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:47726","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:13:10,486 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646590090486","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:47726","multi.service_calls":0,"queuetimems":7,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:13:29,068 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646590109067","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:49650","multi.service_calls":0,"queuetimems":43455,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=152693, hits=144594, hitRatio=94.70%, , cachingAccesses=144713, cachingHits=144285, cachingHitsRatio=99.70%, evictions=13889, evicted=0, evictedPerRun=0.0 2022-03-06 18:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 18:17:08,796 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646590028795","responsesize":"11572","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 7 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:47712","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":7} 2022-03-06 18:17:10,921 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646590330921","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:49680","multi.service_calls":0,"queuetimems":7,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:17:33,263 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646589753261","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 8 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.118:47660","multi.service_calls":0,"queuetimems":22349,"class":"HRegionServer","multi.mutations":8} 2022-03-06 18:17:37,684 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646589757683","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 6 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:49608","multi.service_calls":0,"queuetimems":26770,"class":"HRegionServer","multi.mutations":6} 2022-03-06 18:17:45,267 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646590365266","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:49680","multi.service_calls":0,"queuetimems":34351,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:17:55,458 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646590375458","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:49680","multi.service_calls":0,"queuetimems":44544,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:17:59,588 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646590079585","responsesize":"16528","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 10 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":600003,"client":"172.30.41.118:47712","multi.service_calls":0,"queuetimems":50790,"class":"HRegionServer","multi.mutations":10} 2022-03-06 18:17:59,631 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646590379630","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:47766","multi.service_calls":0,"queuetimems":48709,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:18:02,238 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646590382238","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:49680","multi.service_calls":0,"queuetimems":51324,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:18:09,104 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646590389103","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:47766","multi.service_calls":0,"queuetimems":58182,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:18:10,480 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646590390479","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:47766","multi.service_calls":0,"queuetimems":39307,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:18:10,482 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646590090481","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:49650","multi.service_calls":0,"queuetimems":24869,"class":"HRegionServer","multi.mutations":2} 2022-03-06 18:18:10,487 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646590390487","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:49680","multi.service_calls":0,"queuetimems":36287,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:18:29,068 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646590409068","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:49680","multi.service_calls":0,"queuetimems":52602,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=153460, hits=145361, hitRatio=94.72%, , cachingAccesses=145480, cachingHits=145052, cachingHitsRatio=99.71%, evictions=13919, evicted=0, evictedPerRun=0.0 2022-03-06 18:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 18:22:10,915 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646590330914","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:47766","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-06 18:22:33,263 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646590653263","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:59238","multi.service_calls":0,"queuetimems":24466,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:22:37,685 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646590657684","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:49700","multi.service_calls":0,"queuetimems":28886,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:22:45,613 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646590065612","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 5 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:59202","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":5} 2022-03-06 18:22:59,588 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646590679588","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:47842","multi.service_calls":0,"queuetimems":48666,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:22:59,631 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646590679631","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:47842","multi.service_calls":0,"queuetimems":48709,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:23:09,105 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646590689104","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:49700","multi.service_calls":0,"queuetimems":58182,"class":"HRegionServer","multi.mutations":3} 2022-03-06 18:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=154245, hits=146146, hitRatio=94.75%, , cachingAccesses=146265, cachingHits=145837, cachingHitsRatio=99.71%, evictions=13949, evicted=0, evictedPerRun=0.0 2022-03-06 18:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 18:27:10,916 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646590930916","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:47910","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:27:10,923 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646590630921","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 8 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.118:47842","multi.service_calls":0,"queuetimems":2124,"class":"HRegionServer","multi.mutations":8} 2022-03-06 18:27:59,593 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646590979588","responsesize":"13221","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 8 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300005,"client":"172.31.0.219:49716","multi.service_calls":0,"queuetimems":48672,"class":"HRegionServer","multi.mutations":8} 2022-03-06 18:27:59,633 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646590979631","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.118:47910","multi.service_calls":0,"queuetimems":26367,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:28:02,239 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646590682238","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 5 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:47842","multi.service_calls":0,"queuetimems":51316,"class":"HRegionServer","multi.mutations":5} 2022-03-06 18:28:09,105 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646590989105","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.31.0.219:49682","multi.service_calls":0,"queuetimems":10568,"class":"HRegionServer","multi.mutations":2} 2022-03-06 18:28:10,481 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646590690481","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:47858","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-06 18:28:10,482 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646590690482","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:59238","multi.service_calls":0,"queuetimems":59560,"class":"HRegionServer","multi.mutations":4} 2022-03-06 18:28:10,488 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646590690487","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 5 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:49702","multi.service_calls":0,"queuetimems":6,"class":"HRegionServer","multi.mutations":5} 2022-03-06 18:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=155076, hits=146977, hitRatio=94.78%, , cachingAccesses=147096, cachingHits=146668, cachingHitsRatio=99.71%, evictions=13979, evicted=0, evictedPerRun=0.0 2022-03-06 18:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-06 18:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 18:32:08,799 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646590628797","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 4 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.118:47842","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-06 18:32:10,918 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646591230917","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:47976","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:32:33,265 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646590953264","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 5 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:47910","multi.service_calls":0,"queuetimems":22348,"class":"HRegionServer","multi.mutations":5} 2022-03-06 18:32:37,686 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646590957685","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:47910","multi.service_calls":0,"queuetimems":26769,"class":"HRegionServer","multi.mutations":4} 2022-03-06 18:32:45,268 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646590665267","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 7 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:59238","multi.service_calls":0,"queuetimems":36469,"class":"HRegionServer","multi.mutations":7} 2022-03-06 18:32:45,614 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646590965613","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:49716","multi.service_calls":0,"queuetimems":34697,"class":"HRegionServer","multi.mutations":4} 2022-03-06 18:32:55,461 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646590675458","responsesize":"21487","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 13 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900003,"client":"172.31.0.219:49700","multi.service_calls":0,"queuetimems":46660,"class":"HRegionServer","multi.mutations":13} 2022-03-06 18:32:59,633 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646591279633","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:47976","multi.service_calls":0,"queuetimems":48716,"class":"HRegionServer","multi.mutations":2} 2022-03-06 18:33:02,240 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646591282239","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:49746","multi.service_calls":0,"queuetimems":51322,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:33:10,484 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646591290483","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:47994","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:33:29,070 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646590709069","responsesize":"14879","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 9 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:47858","multi.service_calls":0,"queuetimems":18588,"class":"HRegionServer","multi.mutations":9} 2022-03-06 18:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=155882, hits=147783, hitRatio=94.80%, , cachingAccesses=147902, cachingHits=147474, cachingHitsRatio=99.71%, evictions=14009, evicted=0, evictedPerRun=0.0 2022-03-06 18:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 18:37:10,926 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646591230924","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.155:59276","multi.service_calls":0,"queuetimems":7,"class":"HRegionServer","multi.mutations":2} 2022-03-06 18:37:45,269 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646591565268","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:59292","multi.service_calls":0,"queuetimems":12002,"class":"HRegionServer","multi.mutations":3} 2022-03-06 18:37:45,615 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646591565614","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:48044","multi.service_calls":0,"queuetimems":12348,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:37:59,594 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646591279593","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 5 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:59276","multi.service_calls":0,"queuetimems":48676,"class":"HRegionServer","multi.mutations":5} 2022-03-06 18:37:59,634 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646591579633","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:49784","multi.service_calls":0,"queuetimems":26367,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:38:02,240 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646591582240","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:49784","multi.service_calls":0,"queuetimems":53440,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:38:09,109 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646591289108","responsesize":"16528","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 10 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":600000,"client":"172.31.0.219:49746","multi.service_calls":0,"queuetimems":58190,"class":"HRegionServer","multi.mutations":10} 2022-03-06 18:38:10,483 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646591290482","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 4 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:47994","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-06 18:38:10,485 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646591590484","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:48056","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:38:10,488 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646591290488","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:47994","multi.service_calls":0,"queuetimems":6,"class":"HRegionServer","multi.mutations":6} 2022-03-06 18:38:29,071 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646591609070","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:48056","multi.service_calls":0,"queuetimems":18586,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=156718, hits=148619, hitRatio=94.83%, , cachingAccesses=148738, cachingHits=148310, cachingHitsRatio=99.71%, evictions=14039, evicted=0, evictedPerRun=0.0 2022-03-06 18:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 18:42:08,801 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646591528800","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:59292","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-06 18:42:10,927 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646591830927","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:48104","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:42:33,266 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646591553265","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:48044","multi.service_calls":0,"queuetimems":24465,"class":"HRegionServer","multi.mutations":4} 2022-03-06 18:42:37,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646591557686","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:48044","multi.service_calls":0,"queuetimems":28886,"class":"HRegionServer","multi.mutations":6} 2022-03-06 18:42:55,462 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646591575461","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:49784","multi.service_calls":0,"queuetimems":46661,"class":"HRegionServer","multi.mutations":3} 2022-03-06 18:42:59,596 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646591879595","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:48104","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:42:59,596 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646591879595","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:59310","multi.service_calls":0,"queuetimems":48668,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:42:59,634 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646591879634","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:48104","multi.service_calls":0,"queuetimems":39,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:43:09,109 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646591889109","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:49820","multi.service_calls":0,"queuetimems":9514,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:43:10,485 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646591890485","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:49820","multi.service_calls":0,"queuetimems":59557,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=157548, hits=149449, hitRatio=94.86%, , cachingAccesses=149568, cachingHits=149140, cachingHitsRatio=99.71%, evictions=14069, evicted=0, evictedPerRun=0.0 2022-03-06 18:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 18:47:08,802 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646592128802","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:59328","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:47:10,919 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646591530918","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 5 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:59292","multi.service_calls":0,"queuetimems":2118,"class":"HRegionServer","multi.mutations":5} 2022-03-06 18:47:10,928 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646592130928","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:48174","multi.service_calls":0,"queuetimems":2126,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:47:37,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646592157687","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:59328","multi.service_calls":0,"queuetimems":28885,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:47:45,270 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646591865269","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:48104","multi.service_calls":0,"queuetimems":34342,"class":"HRegionServer","multi.mutations":4} 2022-03-06 18:47:55,463 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646592175462","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:48184","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:47:59,598 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646592179597","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:59330","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:48:10,485 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646591890484","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:59312","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 18:48:10,486 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646592190485","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:59330","multi.service_calls":0,"queuetimems":10888,"class":"HRegionServer","multi.mutations":2} 2022-03-06 18:48:10,489 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646591890488","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:59312","multi.service_calls":0,"queuetimems":4,"class":"HRegionServer","multi.mutations":4} 2022-03-06 18:50:39,629 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 18:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=158301, hits=150202, hitRatio=94.88%, , cachingAccesses=150321, cachingHits=149893, cachingHitsRatio=99.72%, evictions=14099, evicted=0, evictedPerRun=0.0 2022-03-06 18:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 18:52:08,804 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646592428803","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:48238","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:52:33,267 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646592153266","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 5 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:48174","multi.service_calls":0,"queuetimems":24464,"class":"HRegionServer","multi.mutations":5} 2022-03-06 18:52:37,688 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646592457688","responsesize":"6613","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:59360","multi.service_calls":0,"queuetimems":28884,"class":"HRegionServer","multi.mutations":4} 2022-03-06 18:52:59,598 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646592179596","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.155:59330","multi.service_calls":0,"queuetimems":4134,"class":"HRegionServer","multi.mutations":3} 2022-03-06 18:52:59,598 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646592479598","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:48238","multi.service_calls":0,"queuetimems":48669,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:52:59,637 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646592179635","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 4 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.118:48184","multi.service_calls":0,"queuetimems":4173,"class":"HRegionServer","multi.mutations":4} 2022-03-06 18:53:09,110 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646592189109","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 6 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:59330","multi.service_calls":0,"queuetimems":13646,"class":"HRegionServer","multi.mutations":6} 2022-03-06 18:53:10,485 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646592490485","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:59360","multi.service_calls":0,"queuetimems":59556,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:53:29,072 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646591909071","responsesize":"19835","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 12 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.31.0.219:49820","multi.service_calls":0,"queuetimems":78143,"class":"HRegionServer","multi.mutations":12} 2022-03-06 18:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=159115, hits=151016, hitRatio=94.91%, , cachingAccesses=151135, cachingHits=150707, cachingHitsRatio=99.72%, evictions=14129, evicted=0, evictedPerRun=0.0 2022-03-06 18:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 18:57:08,805 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646592728805","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:48282","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 18:57:10,920 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646592430919","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:59360","multi.service_calls":0,"queuetimems":2115,"class":"HRegionServer","multi.mutations":3} 2022-03-06 18:57:37,689 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646592757688","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:59392","multi.service_calls":0,"queuetimems":28883,"class":"HRegionServer","multi.mutations":2} 2022-03-06 18:57:55,464 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646592475463","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 6 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:49874","multi.service_calls":0,"queuetimems":46659,"class":"HRegionServer","multi.mutations":6} 2022-03-06 18:58:09,111 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646592789110","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:59392","multi.service_calls":0,"queuetimems":35842,"class":"HRegionServer","multi.mutations":1} 2022-03-06 18:58:10,486 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646592490486","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.31.0.219:49874","multi.service_calls":0,"queuetimems":59557,"class":"HRegionServer","multi.mutations":2} 2022-03-06 18:58:10,490 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646592490489","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 9 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:48238","multi.service_calls":0,"queuetimems":59560,"class":"HRegionServer","multi.mutations":9} 2022-03-06 19:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=159951, hits=151852, hitRatio=94.94%, , cachingAccesses=151971, cachingHits=151543, cachingHitsRatio=99.72%, evictions=14159, evicted=0, evictedPerRun=0.0 2022-03-06 19:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 19:02:10,921 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646593030920","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:59426","multi.service_calls":0,"queuetimems":2114,"class":"HRegionServer","multi.mutations":2} 2022-03-06 19:02:10,931 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646592430929","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 8 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.118:48238","multi.service_calls":0,"queuetimems":2125,"class":"HRegionServer","multi.mutations":8} 2022-03-06 19:02:33,269 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646592753268","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 4 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:49892","multi.service_calls":0,"queuetimems":24463,"class":"HRegionServer","multi.mutations":4} 2022-03-06 19:02:37,689 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646593057689","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:49920","multi.service_calls":0,"queuetimems":28883,"class":"HRegionServer","multi.mutations":1} 2022-03-06 19:02:45,271 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646592465270","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 7 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:49874","multi.service_calls":0,"queuetimems":36466,"class":"HRegionServer","multi.mutations":7} 2022-03-06 19:02:55,465 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646593075465","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:48342","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 19:02:59,648 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646592779637","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600010,"client":"172.30.41.118:48282","multi.service_calls":0,"queuetimems":26369,"class":"HRegionServer","multi.mutations":2} 2022-03-06 19:03:02,243 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646591882241","responsesize":"18189","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 11 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1500002,"client":"172.30.41.155:59310","multi.service_calls":0,"queuetimems":51314,"class":"HRegionServer","multi.mutations":11} 2022-03-06 19:03:09,112 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646593089112","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:48342","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-06 19:03:29,074 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646592809073","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:48298","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-06 19:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=160698, hits=152599, hitRatio=94.96%, , cachingAccesses=152718, cachingHits=152290, cachingHitsRatio=99.72%, evictions=14189, evicted=0, evictedPerRun=0.0 2022-03-06 19:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 19:07:08,808 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646593028807","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:48336","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":4} 2022-03-06 19:07:10,933 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646593330931","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.31.0.219:49952","multi.service_calls":0,"queuetimems":9,"class":"HRegionServer","multi.mutations":1} 2022-03-06 19:07:33,269 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646593353269","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:48390","multi.service_calls":0,"queuetimems":22347,"class":"HRegionServer","multi.mutations":1} 2022-03-06 19:07:37,690 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646593357689","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:49952","multi.service_calls":0,"queuetimems":26767,"class":"HRegionServer","multi.mutations":1} 2022-03-06 19:07:59,602 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646592779598","responsesize":"16531","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 10 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900004,"client":"172.31.0.219:49892","multi.service_calls":0,"queuetimems":50792,"class":"HRegionServer","multi.mutations":10} 2022-03-06 19:08:10,487 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646593090486","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 4 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:48342","multi.service_calls":0,"queuetimems":15021,"class":"HRegionServer","multi.mutations":4} 2022-03-06 19:08:10,491 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646593090490","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:49926","multi.service_calls":0,"queuetimems":1378,"class":"HRegionServer","multi.mutations":2} 2022-03-06 19:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=161472, hits=153373, hitRatio=94.98%, , cachingAccesses=153492, cachingHits=153064, cachingHitsRatio=99.72%, evictions=14219, evicted=0, evictedPerRun=0.0 2022-03-06 19:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 19:12:10,923 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646593330922","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:59456","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-06 19:12:33,270 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646593653269","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:49988","multi.service_calls":0,"queuetimems":24460,"class":"HRegionServer","multi.mutations":1} 2022-03-06 19:12:45,273 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646593365272","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 9 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:48398","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":9} 2022-03-06 19:12:55,466 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646593375465","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:59460","multi.service_calls":0,"queuetimems":10193,"class":"HRegionServer","multi.mutations":3} 2022-03-06 19:12:59,603 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646593679602","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:48454","multi.service_calls":0,"queuetimems":48668,"class":"HRegionServer","multi.mutations":1} 2022-03-06 19:12:59,603 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646592779598","responsesize":"14882","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 9 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200005,"client":"172.30.41.155:59392","multi.service_calls":0,"queuetimems":26330,"class":"HRegionServer","multi.mutations":9} 2022-03-06 19:13:02,244 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646593382243","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 6 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:49956","multi.service_calls":0,"queuetimems":16971,"class":"HRegionServer","multi.mutations":6} 2022-03-06 19:13:09,113 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646593389112","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:48398","multi.service_calls":0,"queuetimems":13646,"class":"HRegionServer","multi.mutations":2} 2022-03-06 19:13:10,487 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646592790485","responsesize":"24794","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 15 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.118:48282","multi.service_calls":0,"queuetimems":37217,"class":"HRegionServer","multi.mutations":15} 2022-03-06 19:13:10,488 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646593690487","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:49988","multi.service_calls":0,"queuetimems":61678,"class":"HRegionServer","multi.mutations":2} 2022-03-06 19:13:10,491 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646593690491","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:49988","multi.service_calls":0,"queuetimems":59557,"class":"HRegionServer","multi.mutations":1} 2022-03-06 19:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=162332, hits=154233, hitRatio=95.01%, , cachingAccesses=154352, cachingHits=153924, cachingHitsRatio=99.72%, evictions=14249, evicted=0, evictedPerRun=0.0 2022-03-06 19:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 19:17:08,810 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646593628809","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":600001,"client":"172.30.41.118:48454","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 19:17:10,924 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646593930924","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:59494","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 19:17:55,467 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646593975466","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:50016","multi.service_calls":0,"queuetimems":44542,"class":"HRegionServer","multi.mutations":3} 2022-03-06 19:17:59,604 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646593979604","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:59494","multi.service_calls":0,"queuetimems":26333,"class":"HRegionServer","multi.mutations":1} 2022-03-06 19:17:59,650 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646593379648","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.155:59460","multi.service_calls":0,"queuetimems":14376,"class":"HRegionServer","multi.mutations":4} 2022-03-06 19:18:02,244 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646593982244","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:48524","multi.service_calls":0,"queuetimems":28973,"class":"HRegionServer","multi.mutations":1} 2022-03-06 19:18:10,488 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646593990488","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:48524","multi.service_calls":0,"queuetimems":37217,"class":"HRegionServer","multi.mutations":2} 2022-03-06 19:18:29,076 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646593409074","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 8 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.118:48398","multi.service_calls":0,"queuetimems":33608,"class":"HRegionServer","multi.mutations":8} 2022-03-06 19:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=163172, hits=155073, hitRatio=95.04%, , cachingAccesses=155192, cachingHits=154764, cachingHitsRatio=99.72%, evictions=14279, evicted=0, evictedPerRun=0.0 2022-03-06 19:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 19:22:08,811 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646594228811","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300000,"client":"172.30.41.118:48596","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 19:22:10,935 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646593630933","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 5 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.155:59478","multi.service_calls":0,"queuetimems":2124,"class":"HRegionServer","multi.mutations":5} 2022-03-06 19:22:37,692 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646593657690","responsesize":"18183","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 11 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.30.41.118:48454","multi.service_calls":0,"queuetimems":28881,"class":"HRegionServer","multi.mutations":11} 2022-03-06 19:22:45,274 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646593965273","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:48524","multi.service_calls":0,"queuetimems":34349,"class":"HRegionServer","multi.mutations":4} 2022-03-06 19:22:55,468 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646594275468","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:59514","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 19:23:02,245 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646594282244","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:48602","multi.service_calls":0,"queuetimems":6776,"class":"HRegionServer","multi.mutations":2} 2022-03-06 19:23:09,114 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646593989113","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:59494","multi.service_calls":0,"queuetimems":35842,"class":"HRegionServer","multi.mutations":5} 2022-03-06 19:23:10,488 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646593990487","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:50016","multi.service_calls":0,"queuetimems":37216,"class":"HRegionServer","multi.mutations":2} 2022-03-06 19:23:10,488 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646594290488","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:48602","multi.service_calls":0,"queuetimems":15020,"class":"HRegionServer","multi.mutations":1} 2022-03-06 19:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=163983, hits=155884, hitRatio=95.06%, , cachingAccesses=156003, cachingHits=155575, cachingHitsRatio=99.73%, evictions=14309, evicted=0, evictedPerRun=0.0 2022-03-06 19:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 19:27:33,272 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646593953270","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 5 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.155:59494","multi.service_calls":0,"queuetimems":22346,"class":"HRegionServer","multi.mutations":5} 2022-03-06 19:27:45,275 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646594565275","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:48668","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 19:27:55,469 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646594575469","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:59538","multi.service_calls":0,"queuetimems":10194,"class":"HRegionServer","multi.mutations":1} 2022-03-06 19:27:59,606 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646593979603","responsesize":"19835","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 12 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900003,"client":"172.30.41.118:48524","multi.service_calls":0,"queuetimems":48679,"class":"HRegionServer","multi.mutations":12} 2022-03-06 19:27:59,606 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646594279604","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.118:48602","multi.service_calls":0,"queuetimems":4136,"class":"HRegionServer","multi.mutations":4} 2022-03-06 19:28:02,246 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646594582245","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:59538","multi.service_calls":0,"queuetimems":16970,"class":"HRegionServer","multi.mutations":1} 2022-03-06 19:28:09,115 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646594589114","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:59538","multi.service_calls":0,"queuetimems":23839,"class":"HRegionServer","multi.mutations":1} 2022-03-06 19:28:10,492 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646593990491","responsesize":"19835","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 12 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.31.0.219:50016","multi.service_calls":0,"queuetimems":37220,"class":"HRegionServer","multi.mutations":12} 2022-03-06 19:28:11,517 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 19:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=164735, hits=156636, hitRatio=95.08%, , cachingAccesses=156755, cachingHits=156327, cachingHitsRatio=99.73%, evictions=14339, evicted=0, evictedPerRun=0.0 2022-03-06 19:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-06 19:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 19:32:10,926 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646594230924","responsesize":"14879","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 9 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:48596","multi.service_calls":0,"queuetimems":2113,"class":"HRegionServer","multi.mutations":9} 2022-03-06 19:32:33,273 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646594853273","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:50106","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 19:32:37,693 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646594557692","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:59536","multi.service_calls":0,"queuetimems":28880,"class":"HRegionServer","multi.mutations":2} 2022-03-06 19:32:55,470 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646594875470","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:48734","multi.service_calls":0,"queuetimems":22167,"class":"HRegionServer","multi.mutations":2} 2022-03-06 19:32:59,606 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646594879606","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:59558","multi.service_calls":0,"queuetimems":25703,"class":"HRegionServer","multi.mutations":1} 2022-03-06 19:33:02,246 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646594882246","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300000,"client":"172.30.41.118:48740","multi.service_calls":0,"queuetimems":6748,"class":"HRegionServer","multi.mutations":1} 2022-03-06 19:33:10,489 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646594590488","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 8 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:50088","multi.service_calls":0,"queuetimems":25213,"class":"HRegionServer","multi.mutations":8} 2022-03-06 19:33:10,489 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646594590489","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 6 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:48668","multi.service_calls":0,"queuetimems":25214,"class":"HRegionServer","multi.mutations":6} 2022-03-06 19:33:29,077 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646594309076","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:50056","multi.service_calls":0,"queuetimems":33608,"class":"HRegionServer","multi.mutations":3} 2022-03-06 19:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=165508, hits=157409, hitRatio=95.11%, , cachingAccesses=157528, cachingHits=157100, cachingHitsRatio=99.73%, evictions=14369, evicted=0, evictedPerRun=0.0 2022-03-06 19:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 19:37:10,927 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646595130926","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:48790","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-06 19:37:37,693 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646595157693","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:48790","multi.service_calls":0,"queuetimems":26766,"class":"HRegionServer","multi.mutations":1} 2022-03-06 19:37:45,276 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646594865276","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 5 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.31.0.219:50106","multi.service_calls":0,"queuetimems":12003,"class":"HRegionServer","multi.mutations":5} 2022-03-06 19:37:55,471 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646595175470","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:50130","multi.service_calls":0,"queuetimems":44543,"class":"HRegionServer","multi.mutations":1} 2022-03-06 19:37:59,607 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646594879606","responsesize":"16528","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 10 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":600001,"client":"172.31.0.219:50106","multi.service_calls":0,"queuetimems":4135,"class":"HRegionServer","multi.mutations":10} 2022-03-06 19:37:59,652 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646594279650","responsesize":"16534","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 10 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.155:59514","multi.service_calls":0,"queuetimems":4182,"class":"HRegionServer","multi.mutations":10} 2022-03-06 19:38:02,247 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646595182246","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:59574","multi.service_calls":0,"queuetimems":28972,"class":"HRegionServer","multi.mutations":1} 2022-03-06 19:38:10,490 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646595190489","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:50130","multi.service_calls":0,"queuetimems":37215,"class":"HRegionServer","multi.mutations":1} 2022-03-06 19:38:10,493 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646594890492","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:59558","multi.service_calls":0,"queuetimems":36589,"class":"HRegionServer","multi.mutations":5} 2022-03-06 19:38:29,078 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646595209078","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:59576","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 19:39:42,355 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 19:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=166351, hits=158252, hitRatio=95.13%, , cachingAccesses=158371, cachingHits=157943, cachingHitsRatio=99.73%, evictions=14399, evicted=0, evictedPerRun=0.0 2022-03-06 19:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 19:42:08,814 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646594528812","responsesize":"11578","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 7 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200002,"client":"172.30.41.118:48664","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":7} 2022-03-06 19:42:10,928 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646595430928","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:59594","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 19:42:10,938 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646594530936","responsesize":"19838","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 12 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.118:48664","multi.service_calls":0,"queuetimems":2123,"class":"HRegionServer","multi.mutations":12} 2022-03-06 19:42:33,275 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646595153274","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:59574","multi.service_calls":0,"queuetimems":22347,"class":"HRegionServer","multi.mutations":2} 2022-03-06 19:42:37,694 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646595457693","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:48878","multi.service_calls":0,"queuetimems":26765,"class":"HRegionServer","multi.mutations":1} 2022-03-06 19:42:59,609 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646595479607","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.155:59594","multi.service_calls":0,"queuetimems":14330,"class":"HRegionServer","multi.mutations":1} 2022-03-06 19:43:02,247 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646595482247","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300000,"client":"172.30.41.118:48878","multi.service_calls":0,"queuetimems":16970,"class":"HRegionServer","multi.mutations":3} 2022-03-06 19:43:09,116 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646594889115","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 6 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:59558","multi.service_calls":0,"queuetimems":35212,"class":"HRegionServer","multi.mutations":6} 2022-03-06 19:43:10,490 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646595490490","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:48878","multi.service_calls":0,"queuetimems":25213,"class":"HRegionServer","multi.mutations":1} 2022-03-06 19:43:10,494 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646595490493","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:59594","multi.service_calls":0,"queuetimems":25216,"class":"HRegionServer","multi.mutations":1} 2022-03-06 19:43:29,079 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646595509079","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.155:59596","multi.service_calls":0,"queuetimems":29441,"class":"HRegionServer","multi.mutations":2} 2022-03-06 19:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=167204, hits=159105, hitRatio=95.16%, , cachingAccesses=159224, cachingHits=158796, cachingHitsRatio=99.73%, evictions=14429, evicted=0, evictedPerRun=0.0 2022-03-06 19:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 19:47:08,816 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646595728815","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:48934","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 19:47:33,275 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646595753275","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.155:59630","multi.service_calls":0,"queuetimems":24429,"class":"HRegionServer","multi.mutations":2} 2022-03-06 19:47:37,694 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646595757694","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:59628","multi.service_calls":0,"queuetimems":28879,"class":"HRegionServer","multi.mutations":1} 2022-03-06 19:47:45,278 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646595465277","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 6 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:59594","multi.service_calls":0,"queuetimems":34349,"class":"HRegionServer","multi.mutations":6} 2022-03-06 19:47:55,472 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646595475471","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 5 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:48878","multi.service_calls":0,"queuetimems":44543,"class":"HRegionServer","multi.mutations":5} 2022-03-06 19:47:59,610 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646595179607","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900003,"client":"172.30.41.155:59574","multi.service_calls":0,"queuetimems":48680,"class":"HRegionServer","multi.mutations":4} 2022-03-06 19:47:59,610 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646595779610","responsesize":"6613","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300000,"client":"172.30.41.118:48942","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-06 19:48:02,248 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646595782247","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:48942","multi.service_calls":0,"queuetimems":2637,"class":"HRegionServer","multi.mutations":1} 2022-03-06 19:48:09,120 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646595789117","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300003,"client":"172.30.41.155:59634","multi.service_calls":0,"queuetimems":9507,"class":"HRegionServer","multi.mutations":1} 2022-03-06 19:48:10,494 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646595790494","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:48942","multi.service_calls":0,"queuetimems":10884,"class":"HRegionServer","multi.mutations":1} 2022-03-06 19:48:29,080 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646595809079","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:48942","multi.service_calls":0,"queuetimems":19962,"class":"HRegionServer","multi.mutations":1} 2022-03-06 19:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=168003, hits=159904, hitRatio=95.18%, , cachingAccesses=160023, cachingHits=159595, cachingHitsRatio=99.73%, evictions=14459, evicted=0, evictedPerRun=0.0 2022-03-06 19:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 19:52:08,817 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646596028816","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.31.0.219:50078","multi.service_calls":0,"queuetimems":9348,"class":"HRegionServer","multi.mutations":2} 2022-03-06 19:52:10,941 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646595730939","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 9 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.31.0.219:50164","multi.service_calls":0,"queuetimems":2123,"class":"HRegionServer","multi.mutations":9} 2022-03-06 19:52:33,276 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646596053276","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:49000","multi.service_calls":0,"queuetimems":22110,"class":"HRegionServer","multi.mutations":1} 2022-03-06 19:52:37,695 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646596057694","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:49000","multi.service_calls":0,"queuetimems":28877,"class":"HRegionServer","multi.mutations":1} 2022-03-06 19:52:45,278 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646596065278","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:50182","multi.service_calls":0,"queuetimems":30864,"class":"HRegionServer","multi.mutations":1} 2022-03-06 19:52:59,610 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646596079610","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:49000","multi.service_calls":0,"queuetimems":45021,"class":"HRegionServer","multi.mutations":1} 2022-03-06 19:52:59,653 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646595479652","responsesize":"14879","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 9 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:48878","multi.service_calls":0,"queuetimems":48724,"class":"HRegionServer","multi.mutations":9} 2022-03-06 19:53:09,125 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646596089125","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:59670","multi.service_calls":0,"queuetimems":2,"class":"HRegionServer","multi.mutations":1} 2022-03-06 19:53:10,491 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646595190489","responsesize":"24794","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 15 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.31.0.219:50130","multi.service_calls":0,"queuetimems":37214,"class":"HRegionServer","multi.mutations":15} 2022-03-06 19:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=168813, hits=160714, hitRatio=95.20%, , cachingAccesses=160833, cachingHits=160405, cachingHitsRatio=99.73%, evictions=14489, evicted=0, evictedPerRun=0.0 2022-03-06 19:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 19:57:08,819 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646596328818","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:49056","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 19:57:10,942 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646596330942","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:49056","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 19:57:55,473 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646596075472","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:59664","multi.service_calls":0,"queuetimems":46655,"class":"HRegionServer","multi.mutations":2} 2022-03-06 19:58:09,126 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646596389125","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:50228","multi.service_calls":0,"queuetimems":23846,"class":"HRegionServer","multi.mutations":1} 2022-03-06 19:58:10,492 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646595790491","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 5 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:59634","multi.service_calls":0,"queuetimems":10881,"class":"HRegionServer","multi.mutations":5} 2022-03-06 19:58:10,495 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646596090495","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:49012","multi.service_calls":0,"queuetimems":1355,"class":"HRegionServer","multi.mutations":4} 2022-03-06 19:58:29,081 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646596109080","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 5 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":600001,"client":"172.30.41.118:49012","multi.service_calls":0,"queuetimems":19940,"class":"HRegionServer","multi.mutations":5} 2022-03-06 20:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=169612, hits=161513, hitRatio=95.22%, , cachingAccesses=161632, cachingHits=161204, cachingHitsRatio=99.74%, evictions=14519, evicted=0, evictedPerRun=0.0 2022-03-06 20:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 20:02:08,820 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646596628820","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:49116","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:02:10,930 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646595730929","responsesize":"23142","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 14 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200001,"client":"172.30.41.118:48934","multi.service_calls":0,"queuetimems":2114,"class":"HRegionServer","multi.mutations":14} 2022-03-06 20:02:10,943 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646596630943","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:50266","multi.service_calls":0,"queuetimems":2123,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:02:33,277 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646596353276","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:59684","multi.service_calls":0,"queuetimems":22334,"class":"HRegionServer","multi.mutations":4} 2022-03-06 20:02:37,696 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646596357695","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 5 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:59684","multi.service_calls":0,"queuetimems":28877,"class":"HRegionServer","multi.mutations":5} 2022-03-06 20:02:55,473 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646596675473","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300000,"client":"172.30.41.118:49116","multi.service_calls":0,"queuetimems":46653,"class":"HRegionServer","multi.mutations":2} 2022-03-06 20:02:59,613 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646596079610","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 7 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900003,"client":"172.30.41.118:49000","multi.service_calls":0,"queuetimems":50793,"class":"HRegionServer","multi.mutations":7} 2022-03-06 20:02:59,614 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646596379611","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600003,"client":"172.30.41.155:59688","multi.service_calls":0,"queuetimems":14332,"class":"HRegionServer","multi.mutations":3} 2022-03-06 20:03:02,249 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646596082248","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:59664","multi.service_calls":0,"queuetimems":53431,"class":"HRegionServer","multi.mutations":5} 2022-03-06 20:03:10,492 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646596690492","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:49116","multi.service_calls":0,"queuetimems":59549,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:03:10,492 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646596390492","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 4 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.31.0.219:50228","multi.service_calls":0,"queuetimems":25212,"class":"HRegionServer","multi.mutations":4} 2022-03-06 20:03:10,496 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646596690495","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:59706","multi.service_calls":0,"queuetimems":42030,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:03:29,083 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646596709082","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:49128","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=170228, hits=162129, hitRatio=95.24%, , cachingAccesses=162248, cachingHits=161820, cachingHitsRatio=99.74%, evictions=14549, evicted=0, evictedPerRun=0.0 2022-03-06 20:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 20:07:10,944 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646596930943","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:59726","multi.service_calls":0,"queuetimems":2122,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:07:10,945 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646596930944","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:49184","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:07:37,696 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646596957696","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:49182","multi.service_calls":0,"queuetimems":26722,"class":"HRegionServer","multi.mutations":2} 2022-03-06 20:07:45,281 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646596365279","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:49068","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":6} 2022-03-06 20:07:55,475 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646596975474","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:59728","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:07:59,620 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646596979619","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:59728","multi.service_calls":0,"queuetimems":4,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:07:59,660 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646596379653","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900007,"client":"172.30.41.155:59688","multi.service_calls":0,"queuetimems":14374,"class":"HRegionServer","multi.mutations":3} 2022-03-06 20:08:02,250 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646596982249","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:59728","multi.service_calls":0,"queuetimems":6775,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:08:09,127 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646596689126","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:49116","multi.service_calls":0,"queuetimems":58183,"class":"HRegionServer","multi.mutations":4} 2022-03-06 20:08:10,493 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646596990493","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:49190","multi.service_calls":0,"queuetimems":10874,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:08:10,496 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646596990496","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:49190","multi.service_calls":0,"queuetimems":10877,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:11:44,384 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=170884, hits=162785, hitRatio=95.26%, , cachingAccesses=162904, cachingHits=162476, cachingHitsRatio=99.74%, evictions=14579, evicted=0, evictedPerRun=0.0 2022-03-06 20:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 20:12:08,822 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646596928821","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 4 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:59726","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-06 20:12:10,947 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646597230946","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:59736","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-06 20:12:10,947 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646597230946","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:49252","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 20:12:33,278 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646596953277","responsesize":"11572","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 7 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:49184","multi.service_calls":0,"queuetimems":24456,"class":"HRegionServer","multi.mutations":7} 2022-03-06 20:12:45,281 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646597265281","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:59736","multi.service_calls":0,"queuetimems":34335,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:12:59,620 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646597279620","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:50298","multi.service_calls":0,"queuetimems":48674,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:13:09,128 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646597289127","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:49252","multi.service_calls":0,"queuetimems":58180,"class":"HRegionServer","multi.mutations":2} 2022-03-06 20:13:10,493 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646596990492","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:50286","multi.service_calls":0,"queuetimems":15018,"class":"HRegionServer","multi.mutations":2} 2022-03-06 20:13:10,493 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646597290493","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:49252","multi.service_calls":0,"queuetimems":59546,"class":"HRegionServer","multi.mutations":3} 2022-03-06 20:13:10,497 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646597290496","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:49252","multi.service_calls":0,"queuetimems":59549,"class":"HRegionServer","multi.mutations":3} 2022-03-06 20:13:29,084 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646597009083","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 4 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:50286","multi.service_calls":0,"queuetimems":33608,"class":"HRegionServer","multi.mutations":4} 2022-03-06 20:16:44,382 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=171632, hits=163533, hitRatio=95.28%, , cachingAccesses=163652, cachingHits=163224, cachingHitsRatio=99.74%, evictions=14609, evicted=0, evictedPerRun=0.0 2022-03-06 20:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 20:17:08,824 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646597528823","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:59752","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:17:37,697 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646597257697","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:49252","multi.service_calls":0,"queuetimems":26751,"class":"HRegionServer","multi.mutations":2} 2022-03-06 20:17:45,282 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646597565282","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:49322","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 20:17:55,476 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646597275475","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 5 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:49252","multi.service_calls":0,"queuetimems":44529,"class":"HRegionServer","multi.mutations":5} 2022-03-06 20:17:59,617 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646596979613","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900004,"client":"172.30.41.155:59728","multi.service_calls":0,"queuetimems":4139,"class":"HRegionServer","multi.mutations":7} 2022-03-06 20:17:59,621 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646597579620","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:59756","multi.service_calls":0,"queuetimems":14338,"class":"HRegionServer","multi.mutations":2} 2022-03-06 20:17:59,663 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646597279660","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 8 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.31.0.219:50298","multi.service_calls":0,"queuetimems":48714,"class":"HRegionServer","multi.mutations":8} 2022-03-06 20:18:09,128 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646597589128","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:50318","multi.service_calls":0,"queuetimems":23846,"class":"HRegionServer","multi.mutations":3} 2022-03-06 20:18:10,495 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646597590494","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:59756","multi.service_calls":0,"queuetimems":25212,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:18:10,495 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646597590494","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:49322","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:18:29,084 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646597609084","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:50318","multi.service_calls":0,"queuetimems":43802,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=172342, hits=164243, hitRatio=95.30%, , cachingAccesses=164362, cachingHits=163934, cachingHitsRatio=99.74%, evictions=14639, evicted=0, evictedPerRun=0.0 2022-03-06 20:21:48,030 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 20:22:08,825 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646597828825","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:59768","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:22:10,948 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646597530947","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:59752","multi.service_calls":0,"queuetimems":2124,"class":"HRegionServer","multi.mutations":6} 2022-03-06 20:22:10,949 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646597530948","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 8 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:49316","multi.service_calls":0,"queuetimems":2125,"class":"HRegionServer","multi.mutations":8} 2022-03-06 20:22:33,279 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646597553278","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 5 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:49316","multi.service_calls":0,"queuetimems":24454,"class":"HRegionServer","multi.mutations":5} 2022-03-06 20:22:37,698 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646597857697","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:49372","multi.service_calls":0,"queuetimems":28872,"class":"HRegionServer","multi.mutations":3} 2022-03-06 20:22:59,621 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646597879621","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:50336","multi.service_calls":0,"queuetimems":14337,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:23:09,129 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646597889128","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:59772","multi.service_calls":0,"queuetimems":13652,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:23:10,495 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646597890495","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:49390","multi.service_calls":0,"queuetimems":15019,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:23:10,497 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646597590497","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:59756","multi.service_calls":0,"queuetimems":3,"class":"HRegionServer","multi.mutations":3} 2022-03-06 20:23:29,817 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646597909816","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.30.41.118:49398","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 20:25:13,792 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1258ms No GCs detected 2022-03-06 20:25:34,781 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1484ms No GCs detected 2022-03-06 20:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=173141, hits=165042, hitRatio=95.32%, , cachingAccesses=165161, cachingHits=164733, cachingHitsRatio=99.74%, evictions=14669, evicted=0, evictedPerRun=0.0 2022-03-06 20:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 20:27:09,071 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1407ms No GCs detected 2022-03-06 20:27:09,074 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646598128826","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300248,"client":"172.30.41.155:59784","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:27:10,951 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646598130950","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:59784","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:27:37,699 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646598157698","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:49452","multi.service_calls":0,"queuetimems":26748,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:27:45,284 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646597865284","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:59772","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":2} 2022-03-06 20:27:55,477 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646597875476","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:59772","multi.service_calls":0,"queuetimems":10193,"class":"HRegionServer","multi.mutations":4} 2022-03-06 20:27:59,624 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646598179623","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:49456","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":2} 2022-03-06 20:28:02,060 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1394ms No GCs detected 2022-03-06 20:28:04,423 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646597282250","responsesize":"13230","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 8 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1202173,"client":"172.30.41.155:59736","multi.service_calls":0,"queuetimems":51304,"class":"HRegionServer","multi.mutations":8} 2022-03-06 20:28:09,130 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646598189129","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:49456","multi.service_calls":0,"queuetimems":9507,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:28:10,496 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646598190495","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:49456","multi.service_calls":0,"queuetimems":10873,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:28:10,498 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646598190498","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:49456","multi.service_calls":0,"queuetimems":10876,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:28:10,509 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646597890495","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 6 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600014,"client":"172.30.41.118:49390","multi.service_calls":0,"queuetimems":25211,"class":"HRegionServer","multi.mutations":6} 2022-03-06 20:28:29,818 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646598209818","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:49456","multi.service_calls":0,"queuetimems":30196,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=173933, hits=165834, hitRatio=95.34%, , cachingAccesses=165953, cachingHits=165525, cachingHitsRatio=99.74%, evictions=14699, evicted=0, evictedPerRun=0.0 2022-03-06 20:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-06 20:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 20:32:09,106 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646598429105","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.30.41.118:49526","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 20:32:10,951 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646598130950","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:49452","multi.service_calls":0,"queuetimems":2124,"class":"HRegionServer","multi.mutations":3} 2022-03-06 20:32:33,280 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646598153279","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 8 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:49452","multi.service_calls":0,"queuetimems":24453,"class":"HRegionServer","multi.mutations":8} 2022-03-06 20:32:37,699 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646598457699","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:59804","multi.service_calls":0,"queuetimems":26747,"class":"HRegionServer","multi.mutations":2} 2022-03-06 20:32:55,478 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646598475477","responsesize":"8265","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:50370","multi.service_calls":0,"queuetimems":44525,"class":"HRegionServer","multi.mutations":5} 2022-03-06 20:32:59,621 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646597879617","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 7 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900004,"client":"172.30.41.118:49390","multi.service_calls":0,"queuetimems":14334,"class":"HRegionServer","multi.mutations":7} 2022-03-06 20:33:05,326 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646598485326","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:50370","multi.service_calls":0,"queuetimems":20040,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:33:10,499 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646598490498","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:49534","multi.service_calls":0,"queuetimems":10873,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:33:10,510 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646598490509","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:49534","multi.service_calls":0,"queuetimems":10885,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:33:29,819 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646598509818","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:49534","multi.service_calls":0,"queuetimems":30193,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=174709, hits=166610, hitRatio=95.36%, , cachingAccesses=166729, cachingHits=166301, cachingHitsRatio=99.74%, evictions=14729, evicted=0, evictedPerRun=0.0 2022-03-06 20:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 20:37:09,107 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646598729107","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:59818","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:37:10,954 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646598430951","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.118:49528","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-06 20:37:45,286 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646598465285","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 5 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:50370","multi.service_calls":0,"queuetimems":34333,"class":"HRegionServer","multi.mutations":5} 2022-03-06 20:37:55,479 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646598775479","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:49602","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":2} 2022-03-06 20:37:59,622 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646598779621","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:59822","multi.service_calls":0,"queuetimems":4142,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:37:59,625 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646598479624","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:59804","multi.service_calls":0,"queuetimems":14338,"class":"HRegionServer","multi.mutations":3} 2022-03-06 20:37:59,665 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646597879663","responsesize":"24794","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 15 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200001,"client":"172.31.0.219:50336","multi.service_calls":0,"queuetimems":14379,"class":"HRegionServer","multi.mutations":15} 2022-03-06 20:38:05,327 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646598785326","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:59822","multi.service_calls":0,"queuetimems":9847,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:38:10,510 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646598790510","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:59822","multi.service_calls":0,"queuetimems":15031,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:38:29,819 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646598809819","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:59822","multi.service_calls":0,"queuetimems":34340,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:40:18,566 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 20:40:22,266 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1481ms No GCs detected 2022-03-06 20:40:22,385 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 758ms. 2022-03-06 20:40:23,149 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 2 failover attempts. Trying to failover after sleeping for 807ms. 2022-03-06 20:40:58,641 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 20:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=175472, hits=167373, hitRatio=95.38%, , cachingAccesses=167492, cachingHits=167064, cachingHitsRatio=99.74%, evictions=14759, evicted=0, evictedPerRun=0.0 2022-03-06 20:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 20:42:33,282 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646598753281","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 6 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:50386","multi.service_calls":0,"queuetimems":24174,"class":"HRegionServer","multi.mutations":6} 2022-03-06 20:42:45,288 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646599065287","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:49678","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:42:55,480 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646599075479","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:50408","multi.service_calls":0,"queuetimems":10192,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:42:59,665 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646599079665","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:49676","multi.service_calls":0,"queuetimems":14348,"class":"HRegionServer","multi.mutations":2} 2022-03-06 20:43:05,328 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646599085327","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:50408","multi.service_calls":0,"queuetimems":9847,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:43:10,497 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646598490496","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 6 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:50370","multi.service_calls":0,"queuetimems":25210,"class":"HRegionServer","multi.mutations":6} 2022-03-06 20:43:10,500 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646598790499","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 5 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:49602","multi.service_calls":0,"queuetimems":15020,"class":"HRegionServer","multi.mutations":5} 2022-03-06 20:43:29,820 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646599109819","responsesize":"9917","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:49678","multi.service_calls":0,"queuetimems":24491,"class":"HRegionServer","multi.mutations":6} 2022-03-06 20:46:01,585 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 20:46:02,663 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 784ms. 2022-03-06 20:46:45,952 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3100ms No GCs detected 2022-03-06 20:46:45,952 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=176303, hits=168204, hitRatio=95.41%, , cachingAccesses=168323, cachingHits=167895, cachingHitsRatio=99.75%, evictions=14789, evicted=0, evictedPerRun=0.0 2022-03-06 20:46:47,759 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1307ms No GCs detected 2022-03-06 20:46:49,861 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 20:46:53,301 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1402ms No GCs detected 2022-03-06 20:47:04,636 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 20:47:08,992 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1345ms No GCs detected 2022-03-06 20:47:12,164 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646599030954","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 5 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":601210,"client":"172.30.41.118:49672","multi.service_calls":0,"queuetimems":1846,"class":"HRegionServer","multi.mutations":5} 2022-03-06 20:47:10,965 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646598730952","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 6 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":900013,"client":"172.30.41.118:49588","multi.service_calls":0,"queuetimems":1845,"class":"HRegionServer","multi.mutations":6} 2022-03-06 20:47:10,950 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646599029108","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":601842,"client":"172.30.41.118:49672","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-06 20:47:33,283 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646599353282","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:49746","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 20:47:37,701 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646598757699","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.118:49588","multi.service_calls":0,"queuetimems":28592,"class":"HRegionServer","multi.mutations":3} 2022-03-06 20:47:45,288 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646599365288","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:59850","multi.service_calls":0,"queuetimems":12006,"class":"HRegionServer","multi.mutations":2} 2022-03-06 20:47:55,481 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646599375481","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:49746","multi.service_calls":0,"queuetimems":22198,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:47:59,624 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646599079622","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 9 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.31.0.219:50408","multi.service_calls":0,"queuetimems":14335,"class":"HRegionServer","multi.mutations":9} 2022-03-06 20:47:59,629 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646599079625","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 4 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600003,"client":"172.30.41.118:49678","multi.service_calls":0,"queuetimems":14338,"class":"HRegionServer","multi.mutations":4} 2022-03-06 20:48:09,132 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646598489130","responsesize":"11578","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200002,"client":"172.30.41.155:59804","multi.service_calls":0,"queuetimems":23844,"class":"HRegionServer","multi.mutations":7} 2022-03-06 20:48:10,498 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646599390497","responsesize":"11569","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:50442","multi.service_calls":0,"queuetimems":37214,"class":"HRegionServer","multi.mutations":7} 2022-03-06 20:48:10,501 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646599390500","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:59850","multi.service_calls":0,"queuetimems":15019,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:48:29,821 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646599409820","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:49746","multi.service_calls":0,"queuetimems":34339,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=177084, hits=168985, hitRatio=95.43%, , cachingAccesses=169104, cachingHits=168676, cachingHitsRatio=99.75%, evictions=14819, evicted=0, evictedPerRun=0.0 2022-03-06 20:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 20:52:14,506 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646599634505","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:59868","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:52:14,507 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646599634505","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.155:59868","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:52:59,629 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646599679629","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:49794","multi.service_calls":0,"queuetimems":2,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:52:59,630 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646599679624","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300006,"client":"172.30.41.118:49794","multi.service_calls":0,"queuetimems":14335,"class":"HRegionServer","multi.mutations":3} 2022-03-06 20:52:59,667 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646599379665","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.31.0.219:50442","multi.service_calls":0,"queuetimems":26382,"class":"HRegionServer","multi.mutations":4} 2022-03-06 20:53:05,329 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646599385328","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:59850","multi.service_calls":0,"queuetimems":32045,"class":"HRegionServer","multi.mutations":4} 2022-03-06 20:53:09,132 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646599689132","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:49794","multi.service_calls":0,"queuetimems":9505,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:53:10,499 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646599690498","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300001,"client":"172.30.41.118:49794","multi.service_calls":0,"queuetimems":10871,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:53:10,501 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646599690501","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:49794","multi.service_calls":0,"queuetimems":10874,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:53:11,134 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646599090510","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 6 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900624,"client":"172.31.0.219:50408","multi.service_calls":0,"queuetimems":15030,"class":"HRegionServer","multi.mutations":6} 2022-03-06 20:53:29,822 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646599709821","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:49802","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:53:49,441 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2227ms No GCs detected 2022-03-06 20:54:06,266 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1643ms No GCs detected 2022-03-06 20:54:06,272 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 20:54:06,926 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 1 failover attempts. Trying to failover after sleeping for 755ms. 2022-03-06 20:54:08,365 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: tNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 2 failover attempts. Trying to failover after sleeping for 1337ms. 2022-03-06 20:54:14,504 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1201ms No GCs detected 2022-03-06 20:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=177848, hits=169749, hitRatio=95.45%, , cachingAccesses=169868, cachingHits=169440, cachingHitsRatio=99.75%, evictions=14848, evicted=0, evictedPerRun=0.0 2022-03-06 20:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 20:57:14,508 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646599934507","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:49840","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:57:14,508 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646599934507","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:49840","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:57:33,284 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646599653283","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:59868","multi.service_calls":0,"queuetimems":18778,"class":"HRegionServer","multi.mutations":2} 2022-03-06 20:57:37,702 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646599657701","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:50480","multi.service_calls":0,"queuetimems":23195,"class":"HRegionServer","multi.mutations":2} 2022-03-06 20:57:45,290 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646599665289","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 5 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:49794","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":5} 2022-03-06 20:57:59,672 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646599979668","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300004,"client":"172.30.41.118:49856","multi.service_calls":0,"queuetimems":37,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:58:05,330 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646599985329","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:50520","multi.service_calls":0,"queuetimems":5698,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:58:09,133 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646599989132","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:49856","multi.service_calls":0,"queuetimems":9501,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:58:10,499 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646599990499","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:49856","multi.service_calls":0,"queuetimems":10868,"class":"HRegionServer","multi.mutations":1} 2022-03-06 20:58:11,135 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646599991134","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:59890","multi.service_calls":0,"queuetimems":11503,"class":"HRegionServer","multi.mutations":2} 2022-03-06 20:58:29,823 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646600009823","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:59892","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 21:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=178638, hits=170539, hitRatio=95.47%, , cachingAccesses=170658, cachingHits=170230, cachingHitsRatio=99.75%, evictions=14878, evicted=0, evictedPerRun=0.0 2022-03-06 21:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 21:02:14,509 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646600234508","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:49916","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 21:02:15,336 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646599634505","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 8 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900831,"client":"172.30.41.155:59868","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":8} 2022-03-06 21:02:15,337 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646600234509","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300828,"client":"172.30.41.155:59904","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 21:02:59,633 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646599979631","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.118:49856","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":4} 2022-03-06 21:03:00,829 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 21:03:05,330 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646600285330","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:59904","multi.service_calls":0,"queuetimems":32045,"class":"HRegionServer","multi.mutations":1} 2022-03-06 21:03:09,135 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646600289134","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:50540","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 21:03:10,500 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646600290499","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:50540","multi.service_calls":0,"queuetimems":1365,"class":"HRegionServer","multi.mutations":1} 2022-03-06 21:03:10,502 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646599990501","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:59890","multi.service_calls":0,"queuetimems":10870,"class":"HRegionServer","multi.mutations":4} 2022-03-06 21:03:11,136 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646600291135","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.30.41.118:49934","multi.service_calls":0,"queuetimems":1971,"class":"HRegionServer","multi.mutations":2} 2022-03-06 21:03:29,824 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646600309824","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:50540","multi.service_calls":0,"queuetimems":18688,"class":"HRegionServer","multi.mutations":1} 2022-03-06 21:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=179350, hits=171251, hitRatio=95.48%, , cachingAccesses=171370, cachingHits=170942, cachingHitsRatio=99.75%, evictions=14908, evicted=0, evictedPerRun=0.0 2022-03-06 21:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 21:07:15,342 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646600535340","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.118:49988","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-06 21:07:33,287 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646600253284","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 6 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600003,"client":"172.30.41.155:59904","multi.service_calls":0,"queuetimems":18775,"class":"HRegionServer","multi.mutations":6} 2022-03-06 21:07:45,291 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646600265290","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 4 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:50536","multi.service_calls":0,"queuetimems":30780,"class":"HRegionServer","multi.mutations":4} 2022-03-06 21:07:59,632 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646599979631","responsesize":"14879","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 9 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:49856","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":9} 2022-03-06 21:07:59,633 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646600579633","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:59922","multi.service_calls":0,"queuetimems":44296,"class":"HRegionServer","multi.mutations":1} 2022-03-06 21:07:59,673 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646600279672","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:59904","multi.service_calls":0,"queuetimems":26387,"class":"HRegionServer","multi.mutations":4} 2022-03-06 21:08:09,135 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646600589135","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:59922","multi.service_calls":0,"queuetimems":53798,"class":"HRegionServer","multi.mutations":2} 2022-03-06 21:08:29,825 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646600609824","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.30.41.118:50000","multi.service_calls":0,"queuetimems":24463,"class":"HRegionServer","multi.mutations":2} 2022-03-06 21:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=180124, hits=172025, hitRatio=95.50%, , cachingAccesses=172144, cachingHits=171716, cachingHitsRatio=99.75%, evictions=14938, evicted=0, evictedPerRun=0.0 2022-03-06 21:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 21:12:15,339 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646600535338","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:59922","multi.service_calls":0,"queuetimems":3,"class":"HRegionServer","multi.mutations":4} 2022-03-06 21:12:45,292 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646600865291","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:59938","multi.service_calls":0,"queuetimems":29948,"class":"HRegionServer","multi.mutations":1} 2022-03-06 21:12:55,484 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646599675481","responsesize":"24797","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 15 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1500003,"client":"172.31.0.219:50480","multi.service_calls":0,"queuetimems":40975,"class":"HRegionServer","multi.mutations":15} 2022-03-06 21:12:59,674 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646600879673","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:59938","multi.service_calls":0,"queuetimems":26385,"class":"HRegionServer","multi.mutations":1} 2022-03-06 21:13:05,332 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646600585331","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:49998","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-06 21:13:09,136 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646600889135","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:59938","multi.service_calls":0,"queuetimems":35847,"class":"HRegionServer","multi.mutations":1} 2022-03-06 21:13:10,501 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646600590500","responsesize":"11572","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 7 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:50558","multi.service_calls":0,"queuetimems":5169,"class":"HRegionServer","multi.mutations":7} 2022-03-06 21:13:10,503 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646600590502","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:50558","multi.service_calls":0,"queuetimems":5170,"class":"HRegionServer","multi.mutations":4} 2022-03-06 21:13:11,137 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646600591136","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 8 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:49998","multi.service_calls":0,"queuetimems":5805,"class":"HRegionServer","multi.mutations":8} 2022-03-06 21:13:29,827 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646600909826","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:50076","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 21:16:45,163 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=180971, hits=172872, hitRatio=95.52%, , cachingAccesses=172991, cachingHits=172563, cachingHitsRatio=99.75%, evictions=14969, evicted=0, evictedPerRun=0.0 2022-03-06 21:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 21:17:03,249 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 21:17:15,338 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646600535336","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.155:59922","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":5} 2022-03-06 21:17:15,344 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646600835343","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:50050","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-06 21:17:37,705 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646600257702","responsesize":"23142","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 14 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200003,"client":"172.30.41.118:49916","multi.service_calls":0,"queuetimems":23193,"class":"HRegionServer","multi.mutations":14} 2022-03-06 21:17:55,484 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646601175484","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:59960","multi.service_calls":0,"queuetimems":40144,"class":"HRegionServer","multi.mutations":1} 2022-03-06 21:18:05,333 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646601185332","responsesize":"6613","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:59960","multi.service_calls":0,"queuetimems":49992,"class":"HRegionServer","multi.mutations":4} 2022-03-06 21:18:10,532 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646601190531","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300001,"client":"172.30.41.118:50146","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 21:18:15,431 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646601195430","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.155:59966","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 21:18:29,827 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646601209827","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:50134","multi.service_calls":0,"queuetimems":44535,"class":"HRegionServer","multi.mutations":1} 2022-03-06 21:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=181802, hits=173703, hitRatio=95.55%, , cachingAccesses=173822, cachingHits=173394, cachingHitsRatio=99.75%, evictions=14998, evicted=0, evictedPerRun=0.0 2022-03-06 21:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 21:22:15,339 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646601435339","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:50190","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 21:22:15,341 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646601135340","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:50134","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-06 21:22:59,634 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646600879632","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 5 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:59938","multi.service_calls":0,"queuetimems":44289,"class":"HRegionServer","multi.mutations":5} 2022-03-06 21:22:59,636 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646600879633","responsesize":"19835","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 12 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900003,"client":"172.31.0.219:50574","multi.service_calls":0,"queuetimems":44289,"class":"HRegionServer","multi.mutations":12} 2022-03-06 21:22:59,675 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646601179674","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:50590","multi.service_calls":0,"queuetimems":14381,"class":"HRegionServer","multi.mutations":2} 2022-03-06 21:23:10,532 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646601490532","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:59994","multi.service_calls":0,"queuetimems":55193,"class":"HRegionServer","multi.mutations":1} 2022-03-06 21:23:11,138 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646601191137","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:50590","multi.service_calls":0,"queuetimems":55797,"class":"HRegionServer","multi.mutations":3} 2022-03-06 21:23:15,432 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646601495431","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:50208","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 21:23:29,828 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646601509828","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:60002","multi.service_calls":0,"queuetimems":14397,"class":"HRegionServer","multi.mutations":1} 2022-03-06 21:26:26,385 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1042ms No GCs detected 2022-03-06 21:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=182568, hits=174469, hitRatio=95.56%, , cachingAccesses=174588, cachingHits=174160, cachingHitsRatio=99.75%, evictions=15028, evicted=0, evictedPerRun=0.0 2022-03-06 21:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 21:27:15,341 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646601735340","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300001,"client":"172.30.41.118:50260","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 21:27:15,346 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646601435344","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 9 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.118:50190","multi.service_calls":0,"queuetimems":5,"class":"HRegionServer","multi.mutations":9} 2022-03-06 21:27:33,289 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646600853287","responsesize":"21490","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 13 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.118:50050","multi.service_calls":0,"queuetimems":17944,"class":"HRegionServer","multi.mutations":13} 2022-03-06 21:27:37,706 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646601457705","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:50610","multi.service_calls":0,"queuetimems":22366,"class":"HRegionServer","multi.mutations":4} 2022-03-06 21:27:55,486 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646601475484","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:59994","multi.service_calls":0,"queuetimems":40139,"class":"HRegionServer","multi.mutations":3} 2022-03-06 21:27:59,638 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646601779637","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:50628","multi.service_calls":0,"queuetimems":44296,"class":"HRegionServer","multi.mutations":1} 2022-03-06 21:27:59,676 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646601779676","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300000,"client":"172.30.41.118:50268","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 21:28:05,334 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646601485333","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:50190","multi.service_calls":0,"queuetimems":49988,"class":"HRegionServer","multi.mutations":2} 2022-03-06 21:28:09,137 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646601189136","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:59960","multi.service_calls":0,"queuetimems":23843,"class":"HRegionServer","multi.mutations":6} 2022-03-06 21:28:10,533 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646601790532","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:60036","multi.service_calls":0,"queuetimems":10856,"class":"HRegionServer","multi.mutations":2} 2022-03-06 21:28:15,432 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646601795432","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:50268","multi.service_calls":0,"queuetimems":15756,"class":"HRegionServer","multi.mutations":3} 2022-03-06 21:28:29,830 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646601809829","responsesize":"6613","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:50274","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-06 21:30:22,354 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 5154ms No GCs detected 2022-03-06 21:30:26,349 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3494ms No GCs detected 2022-03-06 21:30:35,857 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 21:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=183256, hits=175157, hitRatio=95.58%, , cachingAccesses=175276, cachingHits=174848, cachingHitsRatio=99.76%, evictions=15058, evicted=0, evictedPerRun=0.0 2022-03-06 21:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-06 21:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 21:32:15,343 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646602035343","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:50334","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 21:32:15,347 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646602035347","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:60056","multi.service_calls":0,"queuetimems":4,"class":"HRegionServer","multi.mutations":2} 2022-03-06 21:32:45,296 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646601165292","responsesize":"21490","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 13 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200004,"client":"172.30.41.118:50134","multi.service_calls":0,"queuetimems":29952,"class":"HRegionServer","multi.mutations":13} 2022-03-06 21:32:59,635 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646601779634","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.31.0.219:50628","multi.service_calls":0,"queuetimems":44293,"class":"HRegionServer","multi.mutations":2} 2022-03-06 21:32:59,639 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646602079638","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:50334","multi.service_calls":0,"queuetimems":44291,"class":"HRegionServer","multi.mutations":1} 2022-03-06 21:32:59,732 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646602079676","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300056,"client":"172.30.41.118:50334","multi.service_calls":0,"queuetimems":44329,"class":"HRegionServer","multi.mutations":1} 2022-03-06 21:33:05,335 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646602085334","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:60056","multi.service_calls":0,"queuetimems":49987,"class":"HRegionServer","multi.mutations":1} 2022-03-06 21:33:09,212 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646602089137","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300075,"client":"172.30.41.118:50334","multi.service_calls":0,"queuetimems":53790,"class":"HRegionServer","multi.mutations":1} 2022-03-06 21:33:09,214 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1793ms No GCs detected 2022-03-06 21:33:10,535 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646602090534","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:60060","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 21:33:11,139 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646601791138","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:50628","multi.service_calls":0,"queuetimems":55796,"class":"HRegionServer","multi.mutations":2} 2022-03-06 21:33:15,433 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646602095433","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:50346","multi.service_calls":0,"queuetimems":4899,"class":"HRegionServer","multi.mutations":3} 2022-03-06 21:33:27,232 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 6474ms No GCs detected 2022-03-06 21:33:39,572 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 21:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=183942, hits=175843, hitRatio=95.60%, , cachingAccesses=175962, cachingHits=175534, cachingHitsRatio=99.76%, evictions=15087, evicted=0, evictedPerRun=0.0 2022-03-06 21:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 21:37:15,343 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646601735341","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 8 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.118:50260","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":8} 2022-03-06 21:37:15,345 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646602335344","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:50390","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 21:37:15,350 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646602335348","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.118:50390","multi.service_calls":0,"queuetimems":4,"class":"HRegionServer","multi.mutations":1} 2022-03-06 21:37:37,707 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646602057706","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:50644","multi.service_calls":0,"queuetimems":22363,"class":"HRegionServer","multi.mutations":3} 2022-03-06 21:37:59,732 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646602379732","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:50660","multi.service_calls":0,"queuetimems":44383,"class":"HRegionServer","multi.mutations":2} 2022-03-06 21:38:05,335 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646602385335","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:50660","multi.service_calls":0,"queuetimems":49774,"class":"HRegionServer","multi.mutations":1} 2022-03-06 21:38:09,213 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646602389212","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:50660","multi.service_calls":0,"queuetimems":53863,"class":"HRegionServer","multi.mutations":1} 2022-03-06 21:38:10,535 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646602390535","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:50660","multi.service_calls":0,"queuetimems":54516,"class":"HRegionServer","multi.mutations":1} 2022-03-06 21:38:11,139 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646602391139","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:50660","multi.service_calls":0,"queuetimems":53733,"class":"HRegionServer","multi.mutations":1} 2022-03-06 21:38:14,759 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1322ms No GCs detected 2022-03-06 21:38:15,434 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646602395434","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:50408","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 21:39:12,246 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1392ms No GCs detected 2022-03-06 21:39:13,846 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1100ms No GCs detected 2022-03-06 21:39:36,392 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1226ms No GCs detected 2022-03-06 21:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=184719, hits=176620, hitRatio=95.62%, , cachingAccesses=176739, cachingHits=176311, cachingHitsRatio=99.76%, evictions=15117, evicted=0, evictedPerRun=0.0 2022-03-06 21:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 21:42:15,346 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646602635345","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:60094","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-06 21:42:37,708 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646602657708","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:60094","multi.service_calls":0,"queuetimems":22364,"class":"HRegionServer","multi.mutations":1} 2022-03-06 21:42:59,636 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646602379635","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.31.0.219:50660","multi.service_calls":0,"queuetimems":44290,"class":"HRegionServer","multi.mutations":5} 2022-03-06 21:43:05,336 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646602685336","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:50468","multi.service_calls":0,"queuetimems":5603,"class":"HRegionServer","multi.mutations":1} 2022-03-06 21:43:09,213 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646602689213","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:60098","multi.service_calls":0,"queuetimems":9479,"class":"HRegionServer","multi.mutations":1} 2022-03-06 21:43:15,435 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646602695435","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:50472","multi.service_calls":0,"queuetimems":15671,"class":"HRegionServer","multi.mutations":2} 2022-03-06 21:43:29,832 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646602109830","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 7 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.31.0.219:50648","multi.service_calls":0,"queuetimems":19296,"class":"HRegionServer","multi.mutations":7} 2022-03-06 21:45:04,914 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 21:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=185448, hits=177349, hitRatio=95.63%, , cachingAccesses=177468, cachingHits=177040, cachingHitsRatio=99.76%, evictions=15147, evicted=0, evictedPerRun=0.0 2022-03-06 21:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 21:47:15,345 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646602635344","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 5 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:50456","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":5} 2022-03-06 21:47:15,347 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646602935346","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:50532","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 21:47:37,709 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646602957709","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:50694","multi.service_calls":0,"queuetimems":22362,"class":"HRegionServer","multi.mutations":1} 2022-03-06 21:47:55,488 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646602075486","responsesize":"24794","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 15 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.118:50334","multi.service_calls":0,"queuetimems":40143,"class":"HRegionServer","multi.mutations":15} 2022-03-06 21:47:59,636 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646602979636","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:60116","multi.service_calls":0,"queuetimems":30547,"class":"HRegionServer","multi.mutations":1} 2022-03-06 21:47:59,641 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646602379639","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 5 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.31.0.219:50660","multi.service_calls":0,"queuetimems":44294,"class":"HRegionServer","multi.mutations":5} 2022-03-06 21:48:05,337 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646602985336","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:50532","multi.service_calls":0,"queuetimems":31635,"class":"HRegionServer","multi.mutations":1} 2022-03-06 21:48:09,214 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646602989213","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:50532","multi.service_calls":0,"queuetimems":53867,"class":"HRegionServer","multi.mutations":1} 2022-03-06 21:48:10,536 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646602690535","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:50678","multi.service_calls":0,"queuetimems":10801,"class":"HRegionServer","multi.mutations":3} 2022-03-06 21:48:15,436 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646602995435","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:50694","multi.service_calls":0,"queuetimems":60088,"class":"HRegionServer","multi.mutations":1} 2022-03-06 21:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=186260, hits=178161, hitRatio=95.65%, , cachingAccesses=178280, cachingHits=177852, cachingHitsRatio=99.76%, evictions=15177, evicted=0, evictedPerRun=0.0 2022-03-06 21:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 21:52:15,346 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646603235345","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:50590","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 21:52:15,352 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646602635350","responsesize":"19835","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 12 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900002,"client":"172.30.41.118:50456","multi.service_calls":0,"queuetimems":6,"class":"HRegionServer","multi.mutations":12} 2022-03-06 21:52:33,562 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646602053289","responsesize":"23145","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 14 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1500273,"client":"172.31.0.219:50644","multi.service_calls":0,"queuetimems":17946,"class":"HRegionServer","multi.mutations":14} 2022-03-06 21:52:37,710 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646603257709","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:50590","multi.service_calls":0,"queuetimems":22363,"class":"HRegionServer","multi.mutations":2} 2022-03-06 21:52:45,298 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646602365296","responsesize":"9926","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200002,"client":"172.30.41.155:60076","multi.service_calls":0,"queuetimems":29951,"class":"HRegionServer","multi.mutations":6} 2022-03-06 21:53:10,537 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646603290536","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:50598","multi.service_calls":0,"queuetimems":10894,"class":"HRegionServer","multi.mutations":2} 2022-03-06 21:53:11,142 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646602691139","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 7 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900003,"client":"172.30.41.155:60098","multi.service_calls":0,"queuetimems":11405,"class":"HRegionServer","multi.mutations":7} 2022-03-06 21:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=187055, hits=178956, hitRatio=95.67%, , cachingAccesses=179075, cachingHits=178647, cachingHitsRatio=99.76%, evictions=15207, evicted=0, evictedPerRun=0.0 2022-03-06 21:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 21:57:15,348 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646603235348","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:50590","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":2} 2022-03-06 21:57:33,564 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646603553563","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:50670","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 21:57:45,298 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646603565298","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:60154","multi.service_calls":0,"queuetimems":11735,"class":"HRegionServer","multi.mutations":3} 2022-03-06 21:57:55,489 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646603275489","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:50598","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-06 21:57:59,736 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646602679734","responsesize":"24794","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 15 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.118:50468","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":15} 2022-03-06 21:58:09,542 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646603289541","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:60136","multi.service_calls":0,"queuetimems":14052,"class":"HRegionServer","multi.mutations":4} 2022-03-06 21:58:10,538 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646603590537","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:60154","multi.service_calls":0,"queuetimems":11324,"class":"HRegionServer","multi.mutations":1} 2022-03-06 21:58:11,142 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646603591142","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:60154","multi.service_calls":0,"queuetimems":55795,"class":"HRegionServer","multi.mutations":1} 2022-03-06 21:58:15,437 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646603295436","responsesize":"11572","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:50598","multi.service_calls":0,"queuetimems":15794,"class":"HRegionServer","multi.mutations":7} 2022-03-06 21:58:29,834 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646603009833","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 4 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:60120","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-06 21:58:56,209 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2408ms No GCs detected 2022-03-06 22:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=187873, hits=179774, hitRatio=95.69%, , cachingAccesses=179893, cachingHits=179465, cachingHitsRatio=99.76%, evictions=15237, evicted=0, evictedPerRun=0.0 2022-03-06 22:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 22:02:15,348 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646603535347","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:50670","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 22:02:15,350 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646603835349","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:50728","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 22:02:37,711 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646603557710","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 9 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":600001,"client":"172.30.41.118:50670","multi.service_calls":0,"queuetimems":22363,"class":"HRegionServer","multi.mutations":9} 2022-03-06 22:02:59,638 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646603279636","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:50712","multi.service_calls":0,"queuetimems":4147,"class":"HRegionServer","multi.mutations":4} 2022-03-06 22:02:59,645 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646603279642","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 7 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900003,"client":"172.30.41.118:50598","multi.service_calls":0,"queuetimems":4153,"class":"HRegionServer","multi.mutations":7} 2022-03-06 22:02:59,737 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646603879736","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:50728","multi.service_calls":0,"queuetimems":4246,"class":"HRegionServer","multi.mutations":1} 2022-03-06 22:03:09,542 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646603889542","responsesize":"6613","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:50728","multi.service_calls":0,"queuetimems":14052,"class":"HRegionServer","multi.mutations":4} 2022-03-06 22:03:10,538 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646603890538","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:60168","multi.service_calls":0,"queuetimems":15047,"class":"HRegionServer","multi.mutations":1} 2022-03-06 22:03:11,143 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646603891142","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:60168","multi.service_calls":0,"queuetimems":55793,"class":"HRegionServer","multi.mutations":1} 2022-03-06 22:03:29,836 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646603909835","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:50754","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 22:06:13,060 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1419ms No GCs detected 2022-03-06 22:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=188617, hits=180518, hitRatio=95.71%, , cachingAccesses=180637, cachingHits=180209, cachingHitsRatio=99.76%, evictions=15267, evicted=0, evictedPerRun=0.0 2022-03-06 22:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 22:06:48,795 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 22:07:15,350 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646604135349","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:50798","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 22:07:45,299 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646603865298","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 9 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:50766","multi.service_calls":0,"queuetimems":29947,"class":"HRegionServer","multi.mutations":9} 2022-03-06 22:07:55,491 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646603875490","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:60168","multi.service_calls":0,"queuetimems":40141,"class":"HRegionServer","multi.mutations":3} 2022-03-06 22:07:59,642 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646604179638","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300004,"client":"172.30.41.155:60186","multi.service_calls":0,"queuetimems":44288,"class":"HRegionServer","multi.mutations":1} 2022-03-06 22:07:59,647 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646604179647","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:50808","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-06 22:07:59,737 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646604179737","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:50800","multi.service_calls":0,"queuetimems":91,"class":"HRegionServer","multi.mutations":1} 2022-03-06 22:08:05,339 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646603285337","responsesize":"21490","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 13 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.31.0.219:50712","multi.service_calls":0,"queuetimems":9848,"class":"HRegionServer","multi.mutations":13} 2022-03-06 22:08:09,543 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646604189542","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:50808","multi.service_calls":0,"queuetimems":9896,"class":"HRegionServer","multi.mutations":1} 2022-03-06 22:08:10,539 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646604190538","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:50800","multi.service_calls":0,"queuetimems":10892,"class":"HRegionServer","multi.mutations":1} 2022-03-06 22:08:11,144 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646604191144","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:50808","multi.service_calls":0,"queuetimems":11498,"class":"HRegionServer","multi.mutations":1} 2022-03-06 22:08:29,836 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646604209836","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:50800","multi.service_calls":0,"queuetimems":30190,"class":"HRegionServer","multi.mutations":1} 2022-03-06 22:10:05,184 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1144ms No GCs detected 2022-03-06 22:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=189458, hits=181359, hitRatio=95.73%, , cachingAccesses=181478, cachingHits=181050, cachingHitsRatio=99.76%, evictions=15297, evicted=0, evictedPerRun=0.0 2022-03-06 22:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 22:12:15,355 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646603535352","responsesize":"11578","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 7 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200003,"client":"172.30.41.155:60154","multi.service_calls":0,"queuetimems":5,"class":"HRegionServer","multi.mutations":7} 2022-03-06 22:12:33,566 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646603853564","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 8 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:50728","multi.service_calls":0,"queuetimems":18215,"class":"HRegionServer","multi.mutations":8} 2022-03-06 22:12:45,300 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646604465300","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:50830","multi.service_calls":0,"queuetimems":29949,"class":"HRegionServer","multi.mutations":1} 2022-03-06 22:12:59,649 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646604479647","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.155:60206","multi.service_calls":0,"queuetimems":14347,"class":"HRegionServer","multi.mutations":1} 2022-03-06 22:13:10,706 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646604490705","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300001,"client":"172.30.41.118:50880","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 22:13:11,227 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646604491226","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:50862","multi.service_calls":0,"queuetimems":25926,"class":"HRegionServer","multi.mutations":1} 2022-03-06 22:13:15,438 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646603895437","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:60168","multi.service_calls":0,"queuetimems":19946,"class":"HRegionServer","multi.mutations":7} 2022-03-06 22:13:29,837 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646604509837","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:50882","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 22:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=190151, hits=182052, hitRatio=95.74%, , cachingAccesses=182171, cachingHits=181743, cachingHitsRatio=99.77%, evictions=15327, evicted=0, evictedPerRun=0.0 2022-03-06 22:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 22:17:15,352 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646604135350","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 7 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.155:60186","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":7} 2022-03-06 22:17:15,352 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646604435351","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:50862","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-06 22:17:15,357 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646604735356","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:60222","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 22:17:37,712 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646604157711","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:60186","multi.service_calls":0,"queuetimems":22362,"class":"HRegionServer","multi.mutations":4} 2022-03-06 22:17:45,301 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646604765300","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:50926","multi.service_calls":0,"queuetimems":29944,"class":"HRegionServer","multi.mutations":2} 2022-03-06 22:17:59,643 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646604479642","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:60206","multi.service_calls":0,"queuetimems":44291,"class":"HRegionServer","multi.mutations":3} 2022-03-06 22:17:59,649 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646604779649","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:50868","multi.service_calls":0,"queuetimems":44293,"class":"HRegionServer","multi.mutations":1} 2022-03-06 22:18:05,340 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646604485339","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:50830","multi.service_calls":0,"queuetimems":49988,"class":"HRegionServer","multi.mutations":3} 2022-03-06 22:18:15,439 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646604795438","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:60224","multi.service_calls":0,"queuetimems":4731,"class":"HRegionServer","multi.mutations":1} 2022-03-06 22:18:19,731 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2225ms No GCs detected 2022-03-06 22:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=190976, hits=182877, hitRatio=95.76%, , cachingAccesses=182996, cachingHits=182568, cachingHitsRatio=99.77%, evictions=15357, evicted=0, evictedPerRun=0.0 2022-03-06 22:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 22:22:15,353 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646605035353","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:50998","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 22:22:33,567 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646604753566","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 8 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:50926","multi.service_calls":0,"queuetimems":18210,"class":"HRegionServer","multi.mutations":8} 2022-03-06 22:22:37,713 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646605057712","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.30.41.118:51002","multi.service_calls":0,"queuetimems":22329,"class":"HRegionServer","multi.mutations":2} 2022-03-06 22:22:55,492 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646604475491","responsesize":"18183","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 11 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.30.41.118:50862","multi.service_calls":0,"queuetimems":40140,"class":"HRegionServer","multi.mutations":11} 2022-03-06 22:22:59,651 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646605079650","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:50894","multi.service_calls":0,"queuetimems":44297,"class":"HRegionServer","multi.mutations":3} 2022-03-06 22:22:59,696 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646605079695","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300001,"client":"172.30.41.118:51004","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 22:22:59,740 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646604479738","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 8 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.155:60206","multi.service_calls":0,"queuetimems":14438,"class":"HRegionServer","multi.mutations":8} 2022-03-06 22:23:05,372 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646605085371","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.30.41.118:51008","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 22:23:09,545 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646604489543","responsesize":"18183","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 11 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900002,"client":"172.31.0.219:50830","multi.service_calls":0,"queuetimems":24242,"class":"HRegionServer","multi.mutations":11} 2022-03-06 22:23:10,711 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646604790707","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600004,"client":"172.30.41.118:50944","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-06 22:23:11,228 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646604791227","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 8 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:50944","multi.service_calls":0,"queuetimems":520,"class":"HRegionServer","multi.mutations":8} 2022-03-06 22:23:15,440 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646605095440","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:51014","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 22:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=191718, hits=183619, hitRatio=95.78%, , cachingAccesses=183738, cachingHits=183310, cachingHitsRatio=99.77%, evictions=15387, evicted=0, evictedPerRun=0.0 2022-03-06 22:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 22:27:15,354 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646605335354","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:51074","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 22:27:15,359 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646605035357","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:60238","multi.service_calls":0,"queuetimems":4,"class":"HRegionServer","multi.mutations":3} 2022-03-06 22:27:55,493 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646605375492","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:50912","multi.service_calls":0,"queuetimems":40138,"class":"HRegionServer","multi.mutations":1} 2022-03-06 22:27:59,697 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646605379696","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:60256","multi.service_calls":0,"queuetimems":21982,"class":"HRegionServer","multi.mutations":1} 2022-03-06 22:27:59,741 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646605379740","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:50912","multi.service_calls":0,"queuetimems":44386,"class":"HRegionServer","multi.mutations":2} 2022-03-06 22:28:05,372 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646605385372","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:51074","multi.service_calls":0,"queuetimems":27658,"class":"HRegionServer","multi.mutations":2} 2022-03-06 22:28:09,546 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646605389546","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:51074","multi.service_calls":0,"queuetimems":31832,"class":"HRegionServer","multi.mutations":1} 2022-03-06 22:28:10,711 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646605390711","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:50912","multi.service_calls":0,"queuetimems":32997,"class":"HRegionServer","multi.mutations":1} 2022-03-06 22:28:11,231 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646605391228","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300003,"client":"172.30.41.118:51074","multi.service_calls":0,"queuetimems":33514,"class":"HRegionServer","multi.mutations":1} 2022-03-06 22:28:29,840 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646604809837","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 8 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900003,"client":"172.31.0.219:50876","multi.service_calls":0,"queuetimems":19130,"class":"HRegionServer","multi.mutations":8} 2022-03-06 22:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=192478, hits=184379, hitRatio=95.79%, , cachingAccesses=184498, cachingHits=184070, cachingHitsRatio=99.77%, evictions=15417, evicted=0, evictedPerRun=0.0 2022-03-06 22:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-06 22:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 22:32:59,652 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646605379651","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:60256","multi.service_calls":0,"queuetimems":44297,"class":"HRegionServer","multi.mutations":4} 2022-03-06 22:32:59,747 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646605679746","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:51146","multi.service_calls":0,"queuetimems":44390,"class":"HRegionServer","multi.mutations":3} 2022-03-06 22:32:59,747 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646605679746","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:60272","multi.service_calls":0,"queuetimems":44387,"class":"HRegionServer","multi.mutations":2} 2022-03-06 22:33:09,547 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646605689546","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:51146","multi.service_calls":0,"queuetimems":54187,"class":"HRegionServer","multi.mutations":1} 2022-03-06 22:33:10,712 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646605690712","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:60274","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 22:33:15,441 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646605395440","responsesize":"18180","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 11 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":600001,"client":"172.31.0.219:50912","multi.service_calls":0,"queuetimems":37726,"class":"HRegionServer","multi.mutations":11} 2022-03-06 22:33:29,840 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646605709840","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:51156","multi.service_calls":0,"queuetimems":18608,"class":"HRegionServer","multi.mutations":1} 2022-03-06 22:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=193160, hits=185061, hitRatio=95.81%, , cachingAccesses=185180, cachingHits=184752, cachingHitsRatio=99.77%, evictions=15447, evicted=0, evictedPerRun=0.0 2022-03-06 22:36:48,189 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 22:37:15,356 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646605635355","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:60272","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-06 22:37:15,361 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646605635359","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.155:60272","multi.service_calls":0,"queuetimems":3,"class":"HRegionServer","multi.mutations":4} 2022-03-06 22:37:33,570 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646605353568","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 8 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.118:51074","multi.service_calls":0,"queuetimems":18214,"class":"HRegionServer","multi.mutations":8} 2022-03-06 22:37:37,715 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646605357714","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 7 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:60256","multi.service_calls":0,"queuetimems":22360,"class":"HRegionServer","multi.mutations":7} 2022-03-06 22:37:45,303 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646605065301","responsesize":"14882","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 9 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200002,"client":"172.30.41.155:60238","multi.service_calls":0,"queuetimems":29948,"class":"HRegionServer","multi.mutations":9} 2022-03-06 22:37:55,494 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646605675493","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:51146","multi.service_calls":0,"queuetimems":40137,"class":"HRegionServer","multi.mutations":5} 2022-03-06 22:37:59,747 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646605979747","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:51226","multi.service_calls":0,"queuetimems":91,"class":"HRegionServer","multi.mutations":1} 2022-03-06 22:37:59,748 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646605979747","responsesize":"11569","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:50952","multi.service_calls":0,"queuetimems":93,"class":"HRegionServer","multi.mutations":7} 2022-03-06 22:38:05,373 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646605685373","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 9 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:51146","multi.service_calls":0,"queuetimems":50014,"class":"HRegionServer","multi.mutations":9} 2022-03-06 22:38:09,547 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646605989547","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:51226","multi.service_calls":0,"queuetimems":9891,"class":"HRegionServer","multi.mutations":1} 2022-03-06 22:38:10,713 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646605990713","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:51226","multi.service_calls":0,"queuetimems":11057,"class":"HRegionServer","multi.mutations":1} 2022-03-06 22:38:11,232 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646605691231","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 8 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:51156","multi.service_calls":0,"queuetimems":519,"class":"HRegionServer","multi.mutations":8} 2022-03-06 22:38:15,442 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646605995441","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:51226","multi.service_calls":0,"queuetimems":15693,"class":"HRegionServer","multi.mutations":1} 2022-03-06 22:38:29,841 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646606009840","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:51226","multi.service_calls":0,"queuetimems":30184,"class":"HRegionServer","multi.mutations":1} 2022-03-06 22:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=193913, hits=185814, hitRatio=95.82%, , cachingAccesses=185933, cachingHits=185505, cachingHitsRatio=99.77%, evictions=15477, evicted=0, evictedPerRun=0.0 2022-03-06 22:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 22:42:15,355 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646605035353","responsesize":"29753","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 18 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1500002,"client":"172.30.41.118:50998","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":18} 2022-03-06 22:42:15,362 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646606235361","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:51290","multi.service_calls":0,"queuetimems":3,"class":"HRegionServer","multi.mutations":3} 2022-03-06 22:42:55,495 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646606275494","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:51290","multi.service_calls":0,"queuetimems":40132,"class":"HRegionServer","multi.mutations":1} 2022-03-06 22:42:59,655 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646605979654","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:50952","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-06 22:43:10,714 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646606290713","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.30.41.155:60316","multi.service_calls":0,"queuetimems":10933,"class":"HRegionServer","multi.mutations":2} 2022-03-06 22:43:15,442 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646606295442","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300000,"client":"172.30.41.118:51290","multi.service_calls":0,"queuetimems":12034,"class":"HRegionServer","multi.mutations":1} 2022-03-06 22:43:29,841 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646606309841","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:50966","multi.service_calls":0,"queuetimems":20293,"class":"HRegionServer","multi.mutations":1} 2022-03-06 22:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=194730, hits=186631, hitRatio=95.84%, , cachingAccesses=186750, cachingHits=186322, cachingHitsRatio=99.77%, evictions=15507, evicted=0, evictedPerRun=0.0 2022-03-06 22:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 22:47:15,356 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646606535356","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:51332","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 22:47:15,362 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646606535362","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:50986","multi.service_calls":0,"queuetimems":6,"class":"HRegionServer","multi.mutations":1} 2022-03-06 22:47:33,571 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646606253570","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:60310","multi.service_calls":0,"queuetimems":18212,"class":"HRegionServer","multi.mutations":4} 2022-03-06 22:47:37,716 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646606257715","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:50964","multi.service_calls":0,"queuetimems":22357,"class":"HRegionServer","multi.mutations":3} 2022-03-06 22:47:45,304 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646606265303","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:51290","multi.service_calls":0,"queuetimems":29945,"class":"HRegionServer","multi.mutations":4} 2022-03-06 22:48:09,549 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646606289548","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:60314","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 22:48:10,714 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646606590714","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:60348","multi.service_calls":0,"queuetimems":15218,"class":"HRegionServer","multi.mutations":1} 2022-03-06 22:48:15,443 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646606595442","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:50988","multi.service_calls":0,"queuetimems":19946,"class":"HRegionServer","multi.mutations":2} 2022-03-06 22:49:23,808 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2491ms No GCs detected 2022-03-06 22:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=195593, hits=187494, hitRatio=95.86%, , cachingAccesses=187613, cachingHits=187185, cachingHitsRatio=99.77%, evictions=15537, evicted=0, evictedPerRun=0.0 2022-03-06 22:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 22:52:15,358 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646606835357","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300001,"client":"172.30.41.118:51390","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 22:52:15,359 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646606235358","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:60310","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-06 22:52:15,364 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646606835362","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.118:51390","multi.service_calls":0,"queuetimems":5,"class":"HRegionServer","multi.mutations":2} 2022-03-06 22:52:33,572 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646606853572","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:60384","multi.service_calls":0,"queuetimems":18215,"class":"HRegionServer","multi.mutations":1} 2022-03-06 22:52:55,496 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646606575495","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 8 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":600001,"client":"172.30.41.118:51344","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":8} 2022-03-06 22:52:59,750 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646606279747","responsesize":"18183","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 11 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900003,"client":"172.30.41.118:51290","multi.service_calls":0,"queuetimems":44385,"class":"HRegionServer","multi.mutations":11} 2022-03-06 22:52:59,751 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646606279750","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:50966","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-06 22:53:09,549 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646606289548","responsesize":"16531","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 10 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.31.0.219:50966","multi.service_calls":0,"queuetimems":9798,"class":"HRegionServer","multi.mutations":10} 2022-03-06 22:53:09,549 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646606889549","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:60384","multi.service_calls":0,"queuetimems":35977,"class":"HRegionServer","multi.mutations":3} 2022-03-06 22:53:15,444 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646606895444","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:51410","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-06 22:53:29,843 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646606609842","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:51354","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-06 22:56:39,794 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2616ms No GCs detected 2022-03-06 22:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=196406, hits=188307, hitRatio=95.88%, , cachingAccesses=188426, cachingHits=187998, cachingHitsRatio=99.77%, evictions=15567, evicted=0, evictedPerRun=0.0 2022-03-06 22:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 22:57:02,794 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 22:57:15,360 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646607135359","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:60420","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 22:57:15,364 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646607135364","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:51472","multi.service_calls":0,"queuetimems":5,"class":"HRegionServer","multi.mutations":1} 2022-03-06 22:57:33,573 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646607153572","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:60420","multi.service_calls":0,"queuetimems":18213,"class":"HRegionServer","multi.mutations":1} 2022-03-06 22:58:09,549 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646607189549","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:51028","multi.service_calls":0,"queuetimems":14051,"class":"HRegionServer","multi.mutations":1} 2022-03-06 22:58:10,715 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646606890714","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:60384","multi.service_calls":0,"queuetimems":37142,"class":"HRegionServer","multi.mutations":3} 2022-03-06 22:58:11,234 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646606291232","responsesize":"16534","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 10 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.155:60314","multi.service_calls":0,"queuetimems":1684,"class":"HRegionServer","multi.mutations":10} 2022-03-06 22:58:15,445 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646607195444","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:51028","multi.service_calls":0,"queuetimems":19946,"class":"HRegionServer","multi.mutations":1} 2022-03-06 23:00:03,511 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1039ms No GCs detected 2022-03-06 23:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=197174, hits=189075, hitRatio=95.89%, , cachingAccesses=189194, cachingHits=188766, cachingHitsRatio=99.77%, evictions=15597, evicted=0, evictedPerRun=0.0 2022-03-06 23:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 23:02:15,360 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646607135359","responsesize":"16528","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 10 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":600001,"client":"172.30.41.118:51472","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":10} 2022-03-06 23:02:15,361 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646607435360","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:51540","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 23:02:15,365 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646607435364","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:60442","multi.service_calls":0,"queuetimems":3,"class":"HRegionServer","multi.mutations":1} 2022-03-06 23:02:45,305 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646606865304","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 5 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:60384","multi.service_calls":0,"queuetimems":29946,"class":"HRegionServer","multi.mutations":5} 2022-03-06 23:02:59,657 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646606579655","responsesize":"14882","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 9 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200002,"client":"172.30.41.118:51344","multi.service_calls":0,"queuetimems":4159,"class":"HRegionServer","multi.mutations":9} 2022-03-06 23:02:59,753 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646607179751","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.155:60424","multi.service_calls":0,"queuetimems":4253,"class":"HRegionServer","multi.mutations":5} 2022-03-06 23:03:09,550 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646607489550","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:51044","multi.service_calls":0,"queuetimems":54189,"class":"HRegionServer","multi.mutations":1} 2022-03-06 23:03:09,550 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646607189549","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:60424","multi.service_calls":0,"queuetimems":14051,"class":"HRegionServer","multi.mutations":4} 2022-03-06 23:03:10,716 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646607490716","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:51552","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 23:03:29,844 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646607209843","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 5 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:51028","multi.service_calls":0,"queuetimems":34345,"class":"HRegionServer","multi.mutations":5} 2022-03-06 23:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=197967, hits=189868, hitRatio=95.91%, , cachingAccesses=189987, cachingHits=189559, cachingHitsRatio=99.77%, evictions=15627, evicted=0, evictedPerRun=0.0 2022-03-06 23:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 23:07:37,718 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646606857716","responsesize":"23142","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 14 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.118:51390","multi.service_calls":0,"queuetimems":4144,"class":"HRegionServer","multi.mutations":14} 2022-03-06 23:07:45,306 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646607765305","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:60458","multi.service_calls":0,"queuetimems":29944,"class":"HRegionServer","multi.mutations":1} 2022-03-06 23:07:59,659 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646607779658","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:51616","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 23:07:59,753 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646607179751","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.118:51474","multi.service_calls":0,"queuetimems":4253,"class":"HRegionServer","multi.mutations":4} 2022-03-06 23:08:10,717 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646607790717","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:51616","multi.service_calls":0,"queuetimems":11059,"class":"HRegionServer","multi.mutations":1} 2022-03-06 23:08:11,235 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646607491234","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:60444","multi.service_calls":0,"queuetimems":518,"class":"HRegionServer","multi.mutations":2} 2022-03-06 23:08:29,846 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646607809845","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:51636","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 23:11:44,388 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=198669, hits=190570, hitRatio=95.92%, , cachingAccesses=190689, cachingHits=190261, cachingHitsRatio=99.78%, evictions=15657, evicted=0, evictedPerRun=0.0 2022-03-06 23:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 23:12:33,575 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646607453573","responsesize":"18183","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 11 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900002,"client":"172.30.41.118:51540","multi.service_calls":0,"queuetimems":18212,"class":"HRegionServer","multi.mutations":11} 2022-03-06 23:12:45,307 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646608065306","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:51688","multi.service_calls":0,"queuetimems":7587,"class":"HRegionServer","multi.mutations":1} 2022-03-06 23:12:55,499 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646607175498","responsesize":"18186","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 11 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200001,"client":"172.30.41.118:51474","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":11} 2022-03-06 23:12:59,755 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646608079753","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300002,"client":"172.31.0.219:51088","multi.service_calls":0,"queuetimems":22003,"class":"HRegionServer","multi.mutations":2} 2022-03-06 23:13:10,718 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646608090717","responsesize":"9917","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:51688","multi.service_calls":0,"queuetimems":25410,"class":"HRegionServer","multi.mutations":6} 2022-03-06 23:13:15,446 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646607495445","responsesize":"18183","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 11 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.30.41.118:51552","multi.service_calls":0,"queuetimems":4729,"class":"HRegionServer","multi.mutations":11} 2022-03-06 23:13:29,850 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646608109849","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:60482","multi.service_calls":0,"queuetimems":2,"class":"HRegionServer","multi.mutations":1} 2022-03-06 23:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=199432, hits=191333, hitRatio=95.94%, , cachingAccesses=191452, cachingHits=191024, cachingHitsRatio=99.78%, evictions=15687, evicted=0, evictedPerRun=0.0 2022-03-06 23:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 23:17:15,363 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646607735361","responsesize":"14879","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 9 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:51604","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":9} 2022-03-06 23:17:15,363 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646607735361","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:51604","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-06 23:17:15,367 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646607735365","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 6 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.155:60458","multi.service_calls":0,"queuetimems":4,"class":"HRegionServer","multi.mutations":6} 2022-03-06 23:17:33,576 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646608353576","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:51740","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-06 23:17:37,720 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646608057719","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 4 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:51688","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-06 23:17:45,307 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646608365307","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:60496","multi.service_calls":0,"queuetimems":11731,"class":"HRegionServer","multi.mutations":3} 2022-03-06 23:17:55,500 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646608375500","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:51120","multi.service_calls":0,"queuetimems":21924,"class":"HRegionServer","multi.mutations":1} 2022-03-06 23:17:59,660 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646608079659","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 9 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.31.0.219:51086","multi.service_calls":0,"queuetimems":21940,"class":"HRegionServer","multi.mutations":9} 2022-03-06 23:17:59,755 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646607779753","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 5 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.155:60460","multi.service_calls":0,"queuetimems":95,"class":"HRegionServer","multi.mutations":5} 2022-03-06 23:18:09,551 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646607789550","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 6 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:60460","multi.service_calls":0,"queuetimems":9892,"class":"HRegionServer","multi.mutations":6} 2022-03-06 23:18:10,719 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646608390718","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:60496","multi.service_calls":0,"queuetimems":15218,"class":"HRegionServer","multi.mutations":1} 2022-03-06 23:19:40,347 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1697ms No GCs detected 2022-03-06 23:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=200130, hits=192031, hitRatio=95.95%, , cachingAccesses=192150, cachingHits=191722, cachingHitsRatio=99.78%, evictions=15717, evicted=0, evictedPerRun=0.0 2022-03-06 23:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 23:22:15,364 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646608635363","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:51692","multi.service_calls":0,"queuetimems":13588,"class":"HRegionServer","multi.mutations":2} 2022-03-06 23:22:15,364 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646608635364","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:60518","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-06 23:22:15,376 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646608635367","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300009,"client":"172.30.41.118:51784","multi.service_calls":0,"queuetimems":3,"class":"HRegionServer","multi.mutations":3} 2022-03-06 23:22:33,577 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646608653577","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:51784","multi.service_calls":0,"queuetimems":18213,"class":"HRegionServer","multi.mutations":1} 2022-03-06 23:22:37,720 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646608657720","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:51784","multi.service_calls":0,"queuetimems":20111,"class":"HRegionServer","multi.mutations":1} 2022-03-06 23:22:55,501 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646608675500","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:51162","multi.service_calls":0,"queuetimems":36924,"class":"HRegionServer","multi.mutations":1} 2022-03-06 23:22:59,756 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646608679755","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:60518","multi.service_calls":0,"queuetimems":26178,"class":"HRegionServer","multi.mutations":1} 2022-03-06 23:22:59,756 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646608379755","responsesize":"16528","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 10 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":600001,"client":"172.31.0.219:51120","multi.service_calls":0,"queuetimems":26179,"class":"HRegionServer","multi.mutations":10} 2022-03-06 23:23:09,552 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646608689552","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:51162","multi.service_calls":0,"queuetimems":35974,"class":"HRegionServer","multi.mutations":1} 2022-03-06 23:23:09,552 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646607789550","responsesize":"24794","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 15 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.118:51616","multi.service_calls":0,"queuetimems":9797,"class":"HRegionServer","multi.mutations":15} 2022-03-06 23:23:10,720 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646608690720","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:51798","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 23:23:11,236 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646608091235","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:51688","multi.service_calls":0,"queuetimems":33516,"class":"HRegionServer","multi.mutations":7} 2022-03-06 23:23:15,447 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646608395446","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:51740","multi.service_calls":0,"queuetimems":19946,"class":"HRegionServer","multi.mutations":3} 2022-03-06 23:23:20,373 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1023ms No GCs detected 2022-03-06 23:23:23,640 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1735ms No GCs detected 2022-03-06 23:23:29,851 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646608409850","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:51120","multi.service_calls":0,"queuetimems":56274,"class":"HRegionServer","multi.mutations":3} 2022-03-06 23:26:40,905 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1786ms No GCs detected 2022-03-06 23:26:43,880 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1082ms No GCs detected 2022-03-06 23:26:47,064 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=200854, hits=192755, hitRatio=95.97%, , cachingAccesses=192874, cachingHits=192446, cachingHitsRatio=99.78%, evictions=15747, evicted=0, evictedPerRun=0.0 2022-03-06 23:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 23:27:09,037 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 23:27:37,721 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646608957720","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:51854","multi.service_calls":0,"queuetimems":22344,"class":"HRegionServer","multi.mutations":1} 2022-03-06 23:27:59,661 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646608679660","responsesize":"11572","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 7 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:60518","multi.service_calls":0,"queuetimems":26083,"class":"HRegionServer","multi.mutations":7} 2022-03-06 23:28:10,721 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646608990720","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300001,"client":"172.30.41.118:51858","multi.service_calls":0,"queuetimems":1137,"class":"HRegionServer","multi.mutations":1} 2022-03-06 23:28:11,237 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646608991236","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.30.41.118:51856","multi.service_calls":0,"queuetimems":1653,"class":"HRegionServer","multi.mutations":2} 2022-03-06 23:28:19,324 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3050ms No GCs detected 2022-03-06 23:28:39,335 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 23:29:44,652 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2239ms No GCs detected 2022-03-06 23:29:55,176 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 6016ms No GCs detected 2022-03-06 23:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=201590, hits=193491, hitRatio=95.98%, , cachingAccesses=193610, cachingHits=193182, cachingHitsRatio=99.78%, evictions=15776, evicted=0, evictedPerRun=0.0 2022-03-06 23:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-06 23:31:48,676 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 23:31:48,864 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-06 23:32:15,376 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646608935373","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.155:60534","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-06 23:32:15,377 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646608935375","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 6 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.118:51854","multi.service_calls":0,"queuetimems":2,"class":"HRegionServer","multi.mutations":6} 2022-03-06 23:32:33,578 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646608953577","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:51180","multi.service_calls":0,"queuetimems":18203,"class":"HRegionServer","multi.mutations":3} 2022-03-06 23:32:37,722 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646609257722","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300000,"client":"172.30.41.118:51924","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 23:32:55,502 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646608975502","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:60536","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-06 23:32:59,664 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646609279662","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.155:60554","multi.service_calls":0,"queuetimems":21940,"class":"HRegionServer","multi.mutations":1} 2022-03-06 23:33:10,721 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646609290721","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:51924","multi.service_calls":0,"queuetimems":31453,"class":"HRegionServer","multi.mutations":1} 2022-03-06 23:33:11,238 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646609291237","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:51200","multi.service_calls":0,"queuetimems":33379,"class":"HRegionServer","multi.mutations":1} 2022-03-06 23:33:15,449 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646608995448","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:51182","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-06 23:33:29,852 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646609009851","responsesize":"11572","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:51182","multi.service_calls":0,"queuetimems":14403,"class":"HRegionServer","multi.mutations":7} 2022-03-06 23:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=202397, hits=194298, hitRatio=96.00%, , cachingAccesses=194417, cachingHits=193989, cachingHitsRatio=99.78%, evictions=15806, evicted=0, evictedPerRun=0.0 2022-03-06 23:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 23:37:15,377 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646608935376","responsesize":"14879","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 9 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:51854","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":9} 2022-03-06 23:37:45,309 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646608665308","responsesize":"18186","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 11 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200001,"client":"172.30.41.118:51784","multi.service_calls":0,"queuetimems":29944,"class":"HRegionServer","multi.mutations":11} 2022-03-06 23:37:59,665 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646609579664","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:51218","multi.service_calls":0,"queuetimems":44287,"class":"HRegionServer","multi.mutations":1} 2022-03-06 23:38:09,554 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646608989553","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:51862","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-06 23:38:10,722 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646609590722","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:60572","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 23:38:11,239 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646609591238","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:60572","multi.service_calls":0,"queuetimems":516,"class":"HRegionServer","multi.mutations":3} 2022-03-06 23:38:15,450 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646609595449","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:52002","multi.service_calls":0,"queuetimems":4727,"class":"HRegionServer","multi.mutations":3} 2022-03-06 23:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=203105, hits=195006, hitRatio=96.01%, , cachingAccesses=195125, cachingHits=194697, cachingHitsRatio=99.78%, evictions=15836, evicted=0, evictedPerRun=0.0 2022-03-06 23:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 23:42:15,377 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646609535377","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:51994","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-06 23:42:15,379 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646609835379","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:52062","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 23:42:45,310 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646609865310","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:60590","multi.service_calls":0,"queuetimems":29931,"class":"HRegionServer","multi.mutations":1} 2022-03-06 23:42:55,504 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646609575503","responsesize":"16528","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 10 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":600001,"client":"172.31.0.219:51218","multi.service_calls":0,"queuetimems":40126,"class":"HRegionServer","multi.mutations":10} 2022-03-06 23:42:59,665 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646609879665","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:52062","multi.service_calls":0,"queuetimems":44286,"class":"HRegionServer","multi.mutations":2} 2022-03-06 23:42:59,759 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646608979756","responsesize":"14882","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 9 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200003,"client":"172.30.41.155:60536","multi.service_calls":0,"queuetimems":4254,"class":"HRegionServer","multi.mutations":9} 2022-03-06 23:42:59,759 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646608979756","responsesize":"23142","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 14 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200003,"client":"172.31.0.219:51180","multi.service_calls":0,"queuetimems":44380,"class":"HRegionServer","multi.mutations":14} 2022-03-06 23:43:09,555 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646609889554","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:51236","multi.service_calls":0,"queuetimems":24244,"class":"HRegionServer","multi.mutations":2} 2022-03-06 23:43:10,723 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646609890723","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:52062","multi.service_calls":0,"queuetimems":5442,"class":"HRegionServer","multi.mutations":1} 2022-03-06 23:43:15,450 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646609895450","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:51236","multi.service_calls":0,"queuetimems":7468,"class":"HRegionServer","multi.mutations":1} 2022-03-06 23:43:29,853 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646609609852","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 6 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:52002","multi.service_calls":0,"queuetimems":19130,"class":"HRegionServer","multi.mutations":6} 2022-03-06 23:45:21,922 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-06 23:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=203814, hits=195715, hitRatio=96.03%, , cachingAccesses=195834, cachingHits=195406, cachingHitsRatio=99.78%, evictions=15866, evicted=0, evictedPerRun=0.0 2022-03-06 23:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 23:47:15,379 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646610135378","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:52126","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 23:47:37,724 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646609557723","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:60570","multi.service_calls":0,"queuetimems":22346,"class":"HRegionServer","multi.mutations":4} 2022-03-06 23:47:45,311 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646610165311","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:60608","multi.service_calls":0,"queuetimems":29933,"class":"HRegionServer","multi.mutations":1} 2022-03-06 23:48:01,376 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646610179760","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":301616,"client":"172.30.41.118:52126","multi.service_calls":0,"queuetimems":14449,"class":"HRegionServer","multi.mutations":1} 2022-03-06 23:48:01,376 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2974ms No GCs detected 2022-03-06 23:48:09,555 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646608989553","responsesize":"26449","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 16 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1500002,"client":"172.30.41.118:51862","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":16} 2022-03-06 23:48:10,725 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646610190723","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.155:60608","multi.service_calls":0,"queuetimems":25412,"class":"HRegionServer","multi.mutations":2} 2022-03-06 23:48:15,451 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646610195450","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:51252","multi.service_calls":0,"queuetimems":30138,"class":"HRegionServer","multi.mutations":2} 2022-03-06 23:51:24,030 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2162ms No GCs detected 2022-03-06 23:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=204530, hits=196431, hitRatio=96.04%, , cachingAccesses=196550, cachingHits=196122, cachingHitsRatio=99.78%, evictions=15896, evicted=0, evictedPerRun=0.0 2022-03-06 23:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 23:52:15,379 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646609535377","responsesize":"6622","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200002,"client":"172.30.41.155:60570","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-06 23:52:15,380 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646610435380","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:51270","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 23:52:33,580 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646609553578","responsesize":"23142","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 14 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.118:51994","multi.service_calls":0,"queuetimems":18201,"class":"HRegionServer","multi.mutations":14} 2022-03-06 23:53:01,357 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646610179760","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 4 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":601597,"client":"172.31.0.219:51252","multi.service_calls":0,"queuetimems":44381,"class":"HRegionServer","multi.mutations":4} 2022-03-06 23:53:01,377 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646610481376","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:60626","multi.service_calls":0,"queuetimems":16064,"class":"HRegionServer","multi.mutations":3} 2022-03-06 23:53:09,556 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646610189555","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 9 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:52126","multi.service_calls":0,"queuetimems":24244,"class":"HRegionServer","multi.mutations":9} 2022-03-06 23:53:09,556 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646610489555","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:51270","multi.service_calls":0,"queuetimems":24170,"class":"HRegionServer","multi.mutations":1} 2022-03-06 23:53:10,726 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646610490725","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:52196","multi.service_calls":0,"queuetimems":25413,"class":"HRegionServer","multi.mutations":1} 2022-03-06 23:53:11,240 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646609891239","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:51236","multi.service_calls":0,"queuetimems":55860,"class":"HRegionServer","multi.mutations":3} 2022-03-06 23:55:46,049 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3432ms No GCs detected 2022-03-06 23:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=205228, hits=197129, hitRatio=96.05%, , cachingAccesses=197248, cachingHits=196820, cachingHitsRatio=99.78%, evictions=15926, evicted=0, evictedPerRun=0.0 2022-03-06 23:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-06 23:57:15,381 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646610135379","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 8 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":900001,"client":"172.30.41.118:52126","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":8} 2022-03-06 23:57:45,313 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646610465312","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 6 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:52196","multi.service_calls":0,"queuetimems":29932,"class":"HRegionServer","multi.mutations":6} 2022-03-06 23:57:55,505 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646610175504","responsesize":"14879","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 9 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:51252","multi.service_calls":0,"queuetimems":40125,"class":"HRegionServer","multi.mutations":9} 2022-03-06 23:58:01,358 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646610781358","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:52268","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-06 23:58:11,240 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646610791240","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:51290","multi.service_calls":0,"queuetimems":9882,"class":"HRegionServer","multi.mutations":1} 2022-03-06 23:58:15,452 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646610495451","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:60626","multi.service_calls":0,"queuetimems":30139,"class":"HRegionServer","multi.mutations":2} 2022-03-07 00:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=205993, hits=197894, hitRatio=96.07%, , cachingAccesses=198013, cachingHits=197585, cachingHitsRatio=99.78%, evictions=15956, evicted=0, evictedPerRun=0.0 2022-03-07 00:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 00:02:15,381 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646610735380","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:52258","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 00:02:15,382 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646611035382","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:60658","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 00:02:37,725 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646610457724","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 6 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:52196","multi.service_calls":0,"queuetimems":22344,"class":"HRegionServer","multi.mutations":6} 2022-03-07 00:02:59,667 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646610179665","responsesize":"11578","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 7 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200002,"client":"172.30.41.155:60608","multi.service_calls":0,"queuetimems":44286,"class":"HRegionServer","multi.mutations":7} 2022-03-07 00:03:01,359 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646611081358","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:51308","multi.service_calls":0,"queuetimems":45976,"class":"HRegionServer","multi.mutations":1} 2022-03-07 00:03:01,378 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646610781377","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 4 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:52268","multi.service_calls":0,"queuetimems":19,"class":"HRegionServer","multi.mutations":4} 2022-03-07 00:03:09,558 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646610789556","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.155:60642","multi.service_calls":0,"queuetimems":8198,"class":"HRegionServer","multi.mutations":4} 2022-03-07 00:03:09,558 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646610789556","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.155:60642","multi.service_calls":0,"queuetimems":8198,"class":"HRegionServer","multi.mutations":3} 2022-03-07 00:03:10,726 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646610790726","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.31.0.219:51290","multi.service_calls":0,"queuetimems":9368,"class":"HRegionServer","multi.mutations":3} 2022-03-07 00:03:15,453 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646611095453","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:60660","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 00:03:29,856 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646610209854","responsesize":"9926","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 6 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200002,"client":"172.30.41.118:52142","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":6} 2022-03-07 00:04:32,863 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1281ms No GCs detected 2022-03-07 00:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=206783, hits=198684, hitRatio=96.08%, , cachingAccesses=198803, cachingHits=198375, cachingHitsRatio=99.78%, evictions=15986, evicted=0, evictedPerRun=0.0 2022-03-07 00:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 00:07:13,186 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1074ms No GCs detected 2022-03-07 00:07:15,382 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646610735381","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:60640","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":3} 2022-03-07 00:07:33,583 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646610753580","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 6 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900003,"client":"172.30.41.155:60640","multi.service_calls":0,"queuetimems":18200,"class":"HRegionServer","multi.mutations":6} 2022-03-07 00:07:37,726 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646611357725","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:60672","multi.service_calls":0,"queuetimems":22343,"class":"HRegionServer","multi.mutations":2} 2022-03-07 00:07:55,506 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646611075505","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:60658","multi.service_calls":0,"queuetimems":40123,"class":"HRegionServer","multi.mutations":4} 2022-03-07 00:07:59,667 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646611379667","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:60672","multi.service_calls":0,"queuetimems":44284,"class":"HRegionServer","multi.mutations":1} 2022-03-07 00:08:01,359 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646611381359","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:52402","multi.service_calls":0,"queuetimems":45976,"class":"HRegionServer","multi.mutations":1} 2022-03-07 00:08:09,558 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646611389558","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:60672","multi.service_calls":0,"queuetimems":54175,"class":"HRegionServer","multi.mutations":1} 2022-03-07 00:08:09,558 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646611389558","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:60672","multi.service_calls":0,"queuetimems":54175,"class":"HRegionServer","multi.mutations":1} 2022-03-07 00:08:29,857 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646611409857","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:51342","multi.service_calls":0,"queuetimems":19130,"class":"HRegionServer","multi.mutations":1} 2022-03-07 00:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=207630, hits=199531, hitRatio=96.10%, , cachingAccesses=199650, cachingHits=199222, cachingHitsRatio=99.79%, evictions=16016, evicted=0, evictedPerRun=0.0 2022-03-07 00:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 00:12:15,383 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646611335382","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:52402","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 00:12:15,383 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646611335382","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 4 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:60672","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-07 00:12:33,584 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646611653583","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:52464","multi.service_calls":0,"queuetimems":18200,"class":"HRegionServer","multi.mutations":1} 2022-03-07 00:12:45,314 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646611065313","responsesize":"16531","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 10 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.30.41.118:52322","multi.service_calls":0,"queuetimems":29931,"class":"HRegionServer","multi.mutations":10} 2022-03-07 00:12:59,668 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646611679667","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:51368","multi.service_calls":0,"queuetimems":44283,"class":"HRegionServer","multi.mutations":1} 2022-03-07 00:13:09,559 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646611689558","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:51368","multi.service_calls":0,"queuetimems":35974,"class":"HRegionServer","multi.mutations":1} 2022-03-07 00:13:09,559 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646611689558","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:51368","multi.service_calls":0,"queuetimems":54174,"class":"HRegionServer","multi.mutations":2} 2022-03-07 00:13:10,728 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646611390727","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 5 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:52410","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":5} 2022-03-07 00:13:11,242 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646611091241","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 7 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:52322","multi.service_calls":0,"queuetimems":55859,"class":"HRegionServer","multi.mutations":7} 2022-03-07 00:13:15,456 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646611395453","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600003,"client":"172.30.41.155:60674","multi.service_calls":0,"queuetimems":4726,"class":"HRegionServer","multi.mutations":3} 2022-03-07 00:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=208367, hits=200268, hitRatio=96.11%, , cachingAccesses=200387, cachingHits=199959, cachingHitsRatio=99.79%, evictions=16046, evicted=0, evictedPerRun=0.0 2022-03-07 00:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 00:17:15,384 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646611635383","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 9 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:52464","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":9} 2022-03-07 00:17:33,585 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646611953584","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:51404","multi.service_calls":0,"queuetimems":18200,"class":"HRegionServer","multi.mutations":1} 2022-03-07 00:17:37,727 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646611657726","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:60690","multi.service_calls":0,"queuetimems":22343,"class":"HRegionServer","multi.mutations":4} 2022-03-07 00:17:55,507 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646611675506","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:60690","multi.service_calls":0,"queuetimems":40123,"class":"HRegionServer","multi.mutations":3} 2022-03-07 00:18:01,360 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646611681359","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:52464","multi.service_calls":0,"queuetimems":27775,"class":"HRegionServer","multi.mutations":4} 2022-03-07 00:18:15,456 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646611995456","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:52544","multi.service_calls":0,"queuetimems":5866,"class":"HRegionServer","multi.mutations":2} 2022-03-07 00:18:29,859 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646611709858","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 5 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:52482","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":5} 2022-03-07 00:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=209108, hits=201009, hitRatio=96.13%, , cachingAccesses=201128, cachingHits=200700, cachingHitsRatio=99.79%, evictions=16076, evicted=0, evictedPerRun=0.0 2022-03-07 00:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 00:22:15,385 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646612235385","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300000,"client":"172.30.41.118:52604","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 00:22:15,385 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646611935384","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:52538","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-07 00:22:15,385 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646611935384","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:60708","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-07 00:22:37,727 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646612257727","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:60724","multi.service_calls":0,"queuetimems":4141,"class":"HRegionServer","multi.mutations":1} 2022-03-07 00:23:01,361 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646612281360","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:52604","multi.service_calls":0,"queuetimems":45975,"class":"HRegionServer","multi.mutations":3} 2022-03-07 00:23:01,380 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646611381378","responsesize":"14882","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 9 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200001,"client":"172.30.41.118:52402","multi.service_calls":0,"queuetimems":45996,"class":"HRegionServer","multi.mutations":9} 2022-03-07 00:23:09,562 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646611989560","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.118:52546","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-07 00:23:15,457 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646612295457","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:52618","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 00:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=209834, hits=201735, hitRatio=96.14%, , cachingAccesses=201854, cachingHits=201426, cachingHitsRatio=99.79%, evictions=16106, evicted=0, evictedPerRun=0.0 2022-03-07 00:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 00:27:15,387 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646612535386","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:52672","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 00:27:37,728 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646612557727","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:52672","multi.service_calls":0,"queuetimems":22341,"class":"HRegionServer","multi.mutations":1} 2022-03-07 00:27:45,316 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646611965314","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.155:60708","multi.service_calls":0,"queuetimems":29930,"class":"HRegionServer","multi.mutations":4} 2022-03-07 00:28:01,362 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646612581362","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:60742","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 00:28:01,381 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646612581380","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:52680","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 00:28:09,562 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646611989560","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.155:60710","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":6} 2022-03-07 00:28:11,243 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646611991242","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 6 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:60710","multi.service_calls":0,"queuetimems":1682,"class":"HRegionServer","multi.mutations":6} 2022-03-07 00:28:29,860 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646612309859","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:60726","multi.service_calls":0,"queuetimems":14402,"class":"HRegionServer","multi.mutations":3} 2022-03-07 00:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=210577, hits=202478, hitRatio=96.15%, , cachingAccesses=202597, cachingHits=202169, cachingHitsRatio=99.79%, evictions=16136, evicted=0, evictedPerRun=0.0 2022-03-07 00:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-07 00:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 00:32:15,388 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646612835387","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:52740","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 00:32:37,728 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646612857728","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:52730","multi.service_calls":0,"queuetimems":22310,"class":"HRegionServer","multi.mutations":2} 2022-03-07 00:32:59,670 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646611979668","responsesize":"19838","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 12 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.118:52538","multi.service_calls":0,"queuetimems":44284,"class":"HRegionServer","multi.mutations":12} 2022-03-07 00:33:01,382 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646612881381","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:60756","multi.service_calls":0,"queuetimems":16064,"class":"HRegionServer","multi.mutations":1} 2022-03-07 00:33:09,563 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646612589562","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:52680","multi.service_calls":0,"queuetimems":8200,"class":"HRegionServer","multi.mutations":4} 2022-03-07 00:33:09,563 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646612889562","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:60756","multi.service_calls":0,"queuetimems":24245,"class":"HRegionServer","multi.mutations":2} 2022-03-07 00:33:10,730 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646611990728","responsesize":"19838","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 12 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.118:52546","multi.service_calls":0,"queuetimems":1168,"class":"HRegionServer","multi.mutations":12} 2022-03-07 00:33:11,244 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646612891243","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:51460","multi.service_calls":0,"queuetimems":25926,"class":"HRegionServer","multi.mutations":1} 2022-03-07 00:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=211386, hits=203287, hitRatio=96.17%, , cachingAccesses=203406, cachingHits=202978, cachingHitsRatio=99.79%, evictions=16166, evicted=0, evictedPerRun=0.0 2022-03-07 00:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 00:37:15,387 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646612535386","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:60740","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 00:37:33,588 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646612253586","responsesize":"16534","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 10 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.155:60724","multi.service_calls":0,"queuetimems":18201,"class":"HRegionServer","multi.mutations":10} 2022-03-07 00:37:37,729 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646613157729","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:51478","multi.service_calls":0,"queuetimems":22340,"class":"HRegionServer","multi.mutations":1} 2022-03-07 00:37:45,317 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646612865316","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:60756","multi.service_calls":0,"queuetimems":29929,"class":"HRegionServer","multi.mutations":2} 2022-03-07 00:37:55,509 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646612275507","responsesize":"23142","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 14 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.118:52604","multi.service_calls":0,"queuetimems":21921,"class":"HRegionServer","multi.mutations":14} 2022-03-07 00:38:09,565 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646613189563","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.30.41.155:60780","multi.service_calls":0,"queuetimems":8150,"class":"HRegionServer","multi.mutations":2} 2022-03-07 00:38:15,459 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646612595458","responsesize":"14879","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 9 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:52680","multi.service_calls":0,"queuetimems":14096,"class":"HRegionServer","multi.mutations":9} 2022-03-07 00:38:29,862 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646612909861","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:60758","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 00:38:31,452 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646613211452","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.155:60782","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 00:40:23,980 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2032ms No GCs detected 2022-03-07 00:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=212257, hits=204158, hitRatio=96.18%, , cachingAccesses=204277, cachingHits=203849, cachingHitsRatio=99.79%, evictions=16196, evicted=0, evictedPerRun=0.0 2022-03-07 00:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 00:42:15,388 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646612535386","responsesize":"14882","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 9 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200002,"client":"172.30.41.155:60740","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":9} 2022-03-07 00:42:15,389 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646613435389","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300000,"client":"172.30.41.118:52882","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 00:42:15,389 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646613135389","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 6 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:60776","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":6} 2022-03-07 00:42:37,730 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646613457730","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:60810","multi.service_calls":0,"queuetimems":22341,"class":"HRegionServer","multi.mutations":1} 2022-03-07 00:42:45,318 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646613465317","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:60810","multi.service_calls":0,"queuetimems":29928,"class":"HRegionServer","multi.mutations":3} 2022-03-07 00:42:55,510 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646613475509","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:51494","multi.service_calls":0,"queuetimems":40120,"class":"HRegionServer","multi.mutations":2} 2022-03-07 00:43:01,364 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646612881362","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 8 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.118:52740","multi.service_calls":0,"queuetimems":45974,"class":"HRegionServer","multi.mutations":8} 2022-03-07 00:43:01,383 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646613181382","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 8 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:51478","multi.service_calls":0,"queuetimems":1711,"class":"HRegionServer","multi.mutations":8} 2022-03-07 00:43:10,731 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646613190730","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 5 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:51478","multi.service_calls":0,"queuetimems":55341,"class":"HRegionServer","multi.mutations":5} 2022-03-07 00:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=213051, hits=204952, hitRatio=96.20%, , cachingAccesses=205071, cachingHits=204643, cachingHitsRatio=99.79%, evictions=16226, evicted=0, evictedPerRun=0.0 2022-03-07 00:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 00:47:15,390 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646613735390","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:52942","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 00:47:45,318 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646613765318","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:60846","multi.service_calls":0,"queuetimems":29928,"class":"HRegionServer","multi.mutations":1} 2022-03-07 00:47:55,511 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646613775511","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:60850","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 00:47:59,672 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646613179670","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.118:52808","multi.service_calls":0,"queuetimems":44281,"class":"HRegionServer","multi.mutations":7} 2022-03-07 00:48:01,365 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646613781364","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:52950","multi.service_calls":0,"queuetimems":5853,"class":"HRegionServer","multi.mutations":1} 2022-03-07 00:48:01,383 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646613781383","responsesize":"9917","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:52950","multi.service_calls":0,"queuetimems":5872,"class":"HRegionServer","multi.mutations":6} 2022-03-07 00:48:09,565 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646613189563","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 8 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.118:52808","multi.service_calls":0,"queuetimems":54174,"class":"HRegionServer","multi.mutations":8} 2022-03-07 00:48:15,461 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646613495460","responsesize":"11572","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 7 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:52894","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":7} 2022-03-07 00:48:29,863 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646613509862","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:60814","multi.service_calls":0,"queuetimems":14402,"class":"HRegionServer","multi.mutations":2} 2022-03-07 00:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=213791, hits=205692, hitRatio=96.21%, , cachingAccesses=205811, cachingHits=205383, cachingHitsRatio=99.79%, evictions=16256, evicted=0, evictedPerRun=0.0 2022-03-07 00:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 00:52:15,390 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646613735389","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:52942","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 00:52:33,591 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646613453589","responsesize":"19835","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 12 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900002,"client":"172.30.41.118:52882","multi.service_calls":0,"queuetimems":18200,"class":"HRegionServer","multi.mutations":12} 2022-03-07 00:52:55,513 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646614075512","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:60874","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 00:52:59,673 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646614079672","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:51530","multi.service_calls":0,"queuetimems":44281,"class":"HRegionServer","multi.mutations":1} 2022-03-07 00:53:01,366 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646614081365","responsesize":"13221","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 8 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:51530","multi.service_calls":0,"queuetimems":45973,"class":"HRegionServer","multi.mutations":8} 2022-03-07 00:53:01,384 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646614081383","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:60874","multi.service_calls":0,"queuetimems":5871,"class":"HRegionServer","multi.mutations":1} 2022-03-07 00:53:09,567 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646614089566","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:60874","multi.service_calls":0,"queuetimems":14054,"class":"HRegionServer","multi.mutations":1} 2022-03-07 00:53:15,462 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646614095461","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:51530","multi.service_calls":0,"queuetimems":19949,"class":"HRegionServer","multi.mutations":1} 2022-03-07 00:53:31,454 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646613511452","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 7 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:60814","multi.service_calls":0,"queuetimems":15992,"class":"HRegionServer","multi.mutations":7} 2022-03-07 00:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=214526, hits=206427, hitRatio=96.22%, , cachingAccesses=206546, cachingHits=206118, cachingHitsRatio=99.79%, evictions=16286, evicted=0, evictedPerRun=0.0 2022-03-07 00:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 00:57:15,392 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646614035391","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:53008","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 00:57:37,731 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646613757730","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:51510","multi.service_calls":0,"queuetimems":22341,"class":"HRegionServer","multi.mutations":3} 2022-03-07 00:57:55,513 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646614375513","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:60890","multi.service_calls":0,"queuetimems":40122,"class":"HRegionServer","multi.mutations":1} 2022-03-07 00:57:59,673 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646614379673","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:53086","multi.service_calls":0,"queuetimems":26081,"class":"HRegionServer","multi.mutations":1} 2022-03-07 00:58:01,385 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646614381384","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:53086","multi.service_calls":0,"queuetimems":45993,"class":"HRegionServer","multi.mutations":2} 2022-03-07 00:58:09,567 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646613489565","responsesize":"11578","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 7 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200002,"client":"172.30.41.155:60810","multi.service_calls":0,"queuetimems":31835,"class":"HRegionServer","multi.mutations":7} 2022-03-07 00:58:10,732 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646613790731","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 8 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:60850","multi.service_calls":0,"queuetimems":15220,"class":"HRegionServer","multi.mutations":8} 2022-03-07 00:58:15,493 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646614395493","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:53096","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":2} 2022-03-07 00:58:29,864 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646614109863","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:60874","multi.service_calls":0,"queuetimems":34351,"class":"HRegionServer","multi.mutations":3} 2022-03-07 00:58:31,455 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646614411454","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:53098","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=215263, hits=207164, hitRatio=96.24%, , cachingAccesses=207283, cachingHits=206855, cachingHitsRatio=99.79%, evictions=16316, evicted=0, evictedPerRun=0.0 2022-03-07 01:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 01:02:15,391 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646613735389","responsesize":"23142","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 14 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.118:52942","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":14} 2022-03-07 01:02:15,391 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646614335391","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 5 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:53086","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":5} 2022-03-07 01:02:15,393 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646614635393","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:53150","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 01:02:33,594 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646614353592","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.155:60890","multi.service_calls":0,"queuetimems":18201,"class":"HRegionServer","multi.mutations":4} 2022-03-07 01:02:37,732 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646614657732","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:53152","multi.service_calls":0,"queuetimems":22309,"class":"HRegionServer","multi.mutations":2} 2022-03-07 01:02:55,514 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646614675513","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:60906","multi.service_calls":0,"queuetimems":17781,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:03:01,367 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646614381366","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 5 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:60890","multi.service_calls":0,"queuetimems":27774,"class":"HRegionServer","multi.mutations":5} 2022-03-07 01:03:01,385 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646614681385","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:60906","multi.service_calls":0,"queuetimems":23653,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:03:09,568 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646614389567","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:51550","multi.service_calls":0,"queuetimems":35975,"class":"HRegionServer","multi.mutations":4} 2022-03-07 01:03:10,566 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646614690565","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:60908","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:03:10,734 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646614690733","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:60908","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 01:03:15,494 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646614695493","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:53166","multi.service_calls":0,"queuetimems":4760,"class":"HRegionServer","multi.mutations":2} 2022-03-07 01:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=216014, hits=207915, hitRatio=96.25%, , cachingAccesses=208034, cachingHits=207606, cachingHitsRatio=99.79%, evictions=16346, evicted=0, evictedPerRun=0.0 2022-03-07 01:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 01:07:15,392 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646614935392","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.31.0.219:51406","multi.service_calls":0,"queuetimems":12457,"class":"HRegionServer","multi.mutations":2} 2022-03-07 01:07:15,394 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646614935393","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:53216","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:07:33,507 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1720ms No GCs detected 2022-03-07 01:07:37,733 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646614957732","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:51586","multi.service_calls":0,"queuetimems":22339,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:07:59,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646614679673","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:60906","multi.service_calls":0,"queuetimems":44280,"class":"HRegionServer","multi.mutations":3} 2022-03-07 01:08:01,367 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646614981367","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300000,"client":"172.30.41.118:53216","multi.service_calls":0,"queuetimems":45973,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:08:03,516 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-07 01:08:10,567 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646614990566","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:53232","multi.service_calls":0,"queuetimems":997,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:08:10,734 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646614990734","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:60928","multi.service_calls":0,"queuetimems":1165,"class":"HRegionServer","multi.mutations":2} 2022-03-07 01:08:15,497 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646614995494","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300003,"client":"172.30.41.118:53232","multi.service_calls":0,"queuetimems":5925,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:08:29,866 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646614709865","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:60908","multi.service_calls":0,"queuetimems":19299,"class":"HRegionServer","multi.mutations":4} 2022-03-07 01:09:30,949 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1040ms No GCs detected 2022-03-07 01:10:08,866 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-07 01:10:08,868 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 864ms. 2022-03-07 01:10:10,776 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 2 failover attempts. Trying to failover after sleeping for 1416ms. 2022-03-07 01:10:12,195 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: tNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 3 failover attempts. Trying to failover after sleeping for 2725ms. 2022-03-07 01:10:30,087 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1838ms No GCs detected 2022-03-07 01:10:34,335 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2555ms No GCs detected 2022-03-07 01:10:45,776 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-07 01:11:32,147 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2028ms No GCs detected 2022-03-07 01:11:42,578 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1319ms No GCs detected 2022-03-07 01:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=216738, hits=208639, hitRatio=96.26%, , cachingAccesses=208758, cachingHits=208330, cachingHitsRatio=99.79%, evictions=16376, evicted=0, evictedPerRun=0.0 2022-03-07 01:11:46,879 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-07 01:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 01:12:04,110 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1003ms No GCs detected 2022-03-07 01:12:33,910 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646614953594","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600316,"client":"172.30.41.155:60926","multi.service_calls":0,"queuetimems":18201,"class":"HRegionServer","multi.mutations":3} 2022-03-07 01:12:45,320 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646614065318","responsesize":"26449","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 16 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1500002,"client":"172.30.41.118:53008","multi.service_calls":0,"queuetimems":29927,"class":"HRegionServer","multi.mutations":16} 2022-03-07 01:13:01,368 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646615281367","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:60948","multi.service_calls":0,"queuetimems":1692,"class":"HRegionServer","multi.mutations":2} 2022-03-07 01:13:01,386 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646614981385","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:53216","multi.service_calls":0,"queuetimems":45991,"class":"HRegionServer","multi.mutations":3} 2022-03-07 01:13:09,571 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646614989569","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.155:60928","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 01:13:29,747 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646615309746","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.31.0.219:51632","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 01:13:31,456 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646614711455","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:60908","multi.service_calls":0,"queuetimems":20722,"class":"HRegionServer","multi.mutations":4} 2022-03-07 01:16:19,482 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-07 01:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=217539, hits=209440, hitRatio=96.28%, , cachingAccesses=209559, cachingHits=209131, cachingHitsRatio=99.80%, evictions=16406, evicted=0, evictedPerRun=0.0 2022-03-07 01:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 01:17:15,570 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646614935393","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 8 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900177,"client":"172.30.41.118:53216","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":8} 2022-03-07 01:17:33,913 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646615553911","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.155:60964","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:17:45,354 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646615565353","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:60964","multi.service_calls":0,"queuetimems":11442,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:17:55,516 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646614975514","responsesize":"14879","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 9 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:51586","multi.service_calls":0,"queuetimems":40121,"class":"HRegionServer","multi.mutations":9} 2022-03-07 01:17:59,676 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646615279675","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 8 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:53278","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":8} 2022-03-07 01:18:01,370 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646615581368","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.155:60964","multi.service_calls":0,"queuetimems":23381,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:18:01,387 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646615581386","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.30.41.118:53226","multi.service_calls":0,"queuetimems":28336,"class":"HRegionServer","multi.mutations":2} 2022-03-07 01:18:09,571 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646615589571","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:60964","multi.service_calls":0,"queuetimems":30432,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:18:10,735 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646615290735","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:60948","multi.service_calls":0,"queuetimems":168,"class":"HRegionServer","multi.mutations":2} 2022-03-07 01:18:29,867 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646615309866","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:60948","multi.service_calls":0,"queuetimems":30191,"class":"HRegionServer","multi.mutations":4} 2022-03-07 01:18:46,567 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1004ms No GCs detected 2022-03-07 01:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=218335, hits=210236, hitRatio=96.29%, , cachingAccesses=210355, cachingHits=209927, cachingHitsRatio=99.80%, evictions=16436, evicted=0, evictedPerRun=0.0 2022-03-07 01:21:48,062 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 01:22:15,570 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646615235393","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 5 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900177,"client":"172.30.41.155:60946","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":5} 2022-03-07 01:22:37,987 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646615257733","responsesize":"18183","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 11 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900254,"client":"172.31.0.219:51622","multi.service_calls":0,"queuetimems":22339,"class":"HRegionServer","multi.mutations":11} 2022-03-07 01:22:59,677 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646615879676","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:53376","multi.service_calls":0,"queuetimems":25762,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:23:01,370 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646615881370","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:60980","multi.service_calls":0,"queuetimems":27456,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:23:10,568 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646615290567","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 8 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:53278","multi.service_calls":0,"queuetimems":10892,"class":"HRegionServer","multi.mutations":8} 2022-03-07 01:23:10,736 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646615890736","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:53394","multi.service_calls":0,"queuetimems":1133,"class":"HRegionServer","multi.mutations":2} 2022-03-07 01:23:29,748 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646615609747","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 4 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:53328","multi.service_calls":0,"queuetimems":28378,"class":"HRegionServer","multi.mutations":4} 2022-03-07 01:26:22,983 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1677ms No GCs detected 2022-03-07 01:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=219115, hits=211016, hitRatio=96.30%, , cachingAccesses=211135, cachingHits=210707, cachingHitsRatio=99.80%, evictions=16466, evicted=0, evictedPerRun=0.0 2022-03-07 01:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 01:27:15,569 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646615235394","responsesize":"11578","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200175,"client":"172.30.41.155:60946","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":7} 2022-03-07 01:27:15,571 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646616135571","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:53444","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:27:15,572 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646615835571","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 4 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:53376","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-07 01:27:34,050 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646615853914","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600136,"client":"172.30.41.118:53376","multi.service_calls":0,"queuetimems":18343,"class":"HRegionServer","multi.mutations":6} 2022-03-07 01:27:45,400 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646615865354","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600046,"client":"172.30.41.155:60980","multi.service_calls":0,"queuetimems":29783,"class":"HRegionServer","multi.mutations":2} 2022-03-07 01:27:45,664 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646616165664","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:53458","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:27:59,678 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646616179677","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300001,"client":"172.30.41.118:53458","multi.service_calls":0,"queuetimems":14013,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:28:01,388 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646615881387","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 5 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:51696","multi.service_calls":0,"queuetimems":45816,"class":"HRegionServer","multi.mutations":5} 2022-03-07 01:28:03,101 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646616183101","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:60998","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:28:09,574 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646615889572","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.118:53392","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 01:28:10,569 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646616190569","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:60998","multi.service_calls":0,"queuetimems":7457,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:28:10,737 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646616190736","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:60998","multi.service_calls":0,"queuetimems":2857,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:28:29,748 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646616209748","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:60998","multi.service_calls":0,"queuetimems":19179,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:28:29,868 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646615909867","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:53392","multi.service_calls":0,"queuetimems":20295,"class":"HRegionServer","multi.mutations":2} 2022-03-07 01:28:31,458 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646615611456","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.31.0.219:51668","multi.service_calls":0,"queuetimems":30087,"class":"HRegionServer","multi.mutations":3} 2022-03-07 01:30:49,006 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1451ms No GCs detected 2022-03-07 01:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=219986, hits=211887, hitRatio=96.32%, , cachingAccesses=212006, cachingHits=211578, cachingHitsRatio=99.80%, evictions=16496, evicted=0, evictedPerRun=0.0 2022-03-07 01:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-07 01:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 01:32:15,571 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646616435570","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:53518","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 01:32:15,573 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646616435572","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:53518","multi.service_calls":0,"queuetimems":2,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:32:34,052 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646616454050","responsesize":"9917","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.118:53518","multi.service_calls":0,"queuetimems":18480,"class":"HRegionServer","multi.mutations":6} 2022-03-07 01:32:55,517 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646615875516","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:60980","multi.service_calls":0,"queuetimems":39945,"class":"HRegionServer","multi.mutations":3} 2022-03-07 01:32:59,679 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646616479678","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:51760","multi.service_calls":0,"queuetimems":14012,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:33:09,575 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646616489575","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:51760","multi.service_calls":0,"queuetimems":23909,"class":"HRegionServer","multi.mutations":2} 2022-03-07 01:33:10,570 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646616490569","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:51760","multi.service_calls":0,"queuetimems":10890,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:33:10,737 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646616490737","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:32784","multi.service_calls":0,"queuetimems":11058,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:33:29,869 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646616509869","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:32786","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:33:31,458 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646616511458","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:32786","multi.service_calls":0,"queuetimems":1589,"class":"HRegionServer","multi.mutations":2} 2022-03-07 01:36:05,810 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1527ms No GCs detected 2022-03-07 01:36:07,678 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1368ms No GCs detected 2022-03-07 01:36:26,695 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-07 01:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=220732, hits=212633, hitRatio=96.33%, , cachingAccesses=212752, cachingHits=212324, cachingHitsRatio=99.80%, evictions=16526, evicted=0, evictedPerRun=0.0 2022-03-07 01:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 01:37:15,572 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646616735572","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:53592","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:37:15,573 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646616735573","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:51788","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:37:34,053 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646616754052","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:53592","multi.service_calls":0,"queuetimems":18480,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:37:55,519 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646616775518","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:53604","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 01:38:01,389 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646616481388","responsesize":"16528","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 10 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":600001,"client":"172.30.41.118:53524","multi.service_calls":0,"queuetimems":15723,"class":"HRegionServer","multi.mutations":10} 2022-03-07 01:38:29,749 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646616509748","responsesize":"16528","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 10 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":600001,"client":"172.31.0.219:51760","multi.service_calls":0,"queuetimems":30069,"class":"HRegionServer","multi.mutations":10} 2022-03-07 01:38:29,939 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646616809938","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300001,"client":"172.30.41.118:53614","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:38:33,240 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646616813239","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:32804","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=221528, hits=213429, hitRatio=96.34%, , cachingAccesses=213548, cachingHits=213120, cachingHitsRatio=99.80%, evictions=16556, evicted=0, evictedPerRun=0.0 2022-03-07 01:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 01:42:15,600 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646617035574","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300026,"client":"172.30.41.118:53664","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:42:15,600 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646616435572","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900028,"client":"172.30.41.155:32782","multi.service_calls":0,"queuetimems":2,"class":"HRegionServer","multi.mutations":3} 2022-03-07 01:42:15,601 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646617035574","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300027,"client":"172.30.41.118:53664","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:42:34,106 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646617054053","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300053,"client":"172.31.0.219:51818","multi.service_calls":0,"queuetimems":18479,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:42:45,401 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646616465400","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 5 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:32782","multi.service_calls":0,"queuetimems":29830,"class":"HRegionServer","multi.mutations":5} 2022-03-07 01:42:45,667 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646616465665","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.155:32784","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":7} 2022-03-07 01:42:59,680 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646616779679","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:51792","multi.service_calls":0,"queuetimems":4161,"class":"HRegionServer","multi.mutations":4} 2022-03-07 01:43:03,103 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646616483101","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:51760","multi.service_calls":0,"queuetimems":17435,"class":"HRegionServer","multi.mutations":3} 2022-03-07 01:43:10,640 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646616790570","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 5 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600070,"client":"172.30.41.155:32802","multi.service_calls":0,"queuetimems":15052,"class":"HRegionServer","multi.mutations":5} 2022-03-07 01:43:10,806 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646616790737","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 5 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600069,"client":"172.30.41.118:53604","multi.service_calls":0,"queuetimems":15219,"class":"HRegionServer","multi.mutations":5} 2022-03-07 01:43:29,751 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646617109750","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:53678","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:43:29,940 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646617109940","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:32840","multi.service_calls":0,"queuetimems":189,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:43:33,241 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646617113240","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:32840","multi.service_calls":0,"queuetimems":3489,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=222225, hits=214126, hitRatio=96.36%, , cachingAccesses=214245, cachingHits=213817, cachingHitsRatio=99.80%, evictions=16586, evicted=0, evictedPerRun=0.0 2022-03-07 01:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 01:47:15,603 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646617335603","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:51840","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:47:15,643 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646617335642","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:53714","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:47:34,108 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646617354106","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.118:53714","multi.service_calls":0,"queuetimems":18465,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:47:45,402 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646617365402","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:32866","multi.service_calls":0,"queuetimems":29761,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:47:45,667 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646617365667","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.155:32868","multi.service_calls":0,"queuetimems":29901,"class":"HRegionServer","multi.mutations":2} 2022-03-07 01:47:55,521 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646617075519","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:53670","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 01:47:59,684 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646617379682","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.155:32870","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:48:03,103 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646617383103","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:32870","multi.service_calls":0,"queuetimems":3422,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:48:09,578 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646616789575","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 5 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900003,"client":"172.30.41.155:32802","multi.service_calls":0,"queuetimems":14057,"class":"HRegionServer","multi.mutations":5} 2022-03-07 01:48:10,641 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646617390641","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:51842","multi.service_calls":0,"queuetimems":10960,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:48:10,807 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646617390806","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:32870","multi.service_calls":0,"queuetimems":11125,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:48:30,132 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646617409940","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300192,"client":"172.31.0.219:51842","multi.service_calls":0,"queuetimems":30259,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:48:30,133 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646617409751","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300381,"client":"172.30.41.155:32870","multi.service_calls":0,"queuetimems":30070,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:48:31,890 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1616ms No GCs detected 2022-03-07 01:48:33,241 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646617413241","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:51842","multi.service_calls":0,"queuetimems":33560,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:49:38,529 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4110ms No GCs detected 2022-03-07 01:50:13,599 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 8049ms No GCs detected 2022-03-07 01:50:13,687 INFO [regionserver/cvp328:16201.Chore.3] hbase.ScheduledChore: Chore: CompactionChecker missed its start time 2022-03-07 01:50:13,687 INFO [regionserver/cvp328:16201.Chore.3] hbase.ScheduledChore: Chore: MemstoreFlusherChore missed its start time 2022-03-07 01:50:24,170 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3959ms No GCs detected 2022-03-07 01:50:30,173 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-07 01:50:36,815 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1133ms No GCs detected 2022-03-07 01:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=222935, hits=214836, hitRatio=96.37%, , cachingAccesses=214955, cachingHits=214527, cachingHitsRatio=99.80%, evictions=16616, evicted=0, evictedPerRun=0.0 2022-03-07 01:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 01:52:34,108 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646617654108","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:51866","multi.service_calls":0,"queuetimems":18503,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:52:45,403 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646617665402","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:51866","multi.service_calls":0,"queuetimems":29797,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:52:55,521 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646617675521","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:51866","multi.service_calls":0,"queuetimems":39916,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:52:59,685 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646617679684","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:51866","multi.service_calls":0,"queuetimems":44079,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:53:01,391 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646617081389","responsesize":"16531","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 10 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900002,"client":"172.30.41.118:53670","multi.service_calls":0,"queuetimems":5869,"class":"HRegionServer","multi.mutations":10} 2022-03-07 01:53:03,275 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646617683103","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300171,"client":"172.31.0.219:51866","multi.service_calls":0,"queuetimems":47498,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:53:09,580 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646617689578","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.31.0.219:51866","multi.service_calls":0,"queuetimems":53973,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:53:10,642 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646617690641","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:51866","multi.service_calls":0,"queuetimems":55036,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:53:10,808 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646617690807","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:51866","multi.service_calls":0,"queuetimems":55202,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:53:30,165 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646617710165","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.31.0.219:51874","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 01:53:38,326 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1498ms No GCs detected 2022-03-07 01:54:01,416 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-07 01:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=223726, hits=215627, hitRatio=96.38%, , cachingAccesses=215746, cachingHits=215318, cachingHitsRatio=99.80%, evictions=16645, evicted=0, evictedPerRun=0.0 2022-03-07 01:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 01:57:15,605 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646617635604","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 4 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:32888","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-07 01:57:45,668 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646617665667","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:32888","multi.service_calls":0,"queuetimems":30063,"class":"HRegionServer","multi.mutations":4} 2022-03-07 01:57:55,522 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646617975522","responsesize":"11569","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:51900","multi.service_calls":0,"queuetimems":21412,"class":"HRegionServer","multi.mutations":7} 2022-03-07 01:58:09,580 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646617989580","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:51900","multi.service_calls":0,"queuetimems":35470,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:58:10,642 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646617990642","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:51900","multi.service_calls":0,"queuetimems":15120,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:58:10,808 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646617990808","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:32904","multi.service_calls":0,"queuetimems":15286,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:58:30,167 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646618010167","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:53836","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 01:58:31,902 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646617711901","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:53788","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":5} 2022-03-07 01:58:33,266 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646617713266","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:32890","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":2} 2022-03-07 01:58:40,406 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1423ms No GCs detected 2022-03-07 01:58:58,602 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1692ms No GCs detected 2022-03-07 01:59:00,520 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1418ms No GCs detected 2022-03-07 01:59:04,189 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-07 01:59:04,192 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 427ms. 2022-03-07 01:59:04,626 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: tNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 2 failover attempts. Trying to failover after sleeping for 1052ms. 2022-03-07 01:59:18,959 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1428ms No GCs detected 2022-03-07 01:59:20,648 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1188ms No GCs detected 2022-03-07 01:59:35,238 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3358ms No GCs detected 2022-03-07 01:59:36,241 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-07 02:00:08,201 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-07 02:00:43,788 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1348ms No GCs detected 2022-03-07 02:01:04,673 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2040ms No GCs detected 2022-03-07 02:01:06,487 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1314ms No GCs detected 2022-03-07 02:01:10,167 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1179ms No GCs detected 2022-03-07 02:01:15,637 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1818ms No GCs detected 2022-03-07 02:01:39,544 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3061ms No GCs detected 2022-03-07 02:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=224526, hits=216427, hitRatio=96.39%, , cachingAccesses=216546, cachingHits=216118, cachingHitsRatio=99.80%, evictions=16675, evicted=0, evictedPerRun=0.0 2022-03-07 02:01:49,505 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1202ms No GCs detected 2022-03-07 02:01:49,505 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 02:02:15,644 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646617635643","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 8 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":900001,"client":"172.30.41.118:53776","multi.service_calls":0,"queuetimems":38,"class":"HRegionServer","multi.mutations":8} 2022-03-07 02:02:34,111 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646617954109","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.155:32904","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 02:02:36,622 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1179ms No GCs detected 2022-03-07 02:02:45,404 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646617965403","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:53822","multi.service_calls":0,"queuetimems":11294,"class":"HRegionServer","multi.mutations":4} 2022-03-07 02:02:55,554 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646618275554","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:53890","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":2} 2022-03-07 02:03:02,057 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646617981391","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 5 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600666,"client":"172.30.41.155:32904","multi.service_calls":0,"queuetimems":5869,"class":"HRegionServer","multi.mutations":5} 2022-03-07 02:03:12,361 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3130ms No GCs detected 2022-03-07 02:03:12,476 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646618290642","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":301834,"client":"172.31.0.219:51938","multi.service_calls":0,"queuetimems":1061,"class":"HRegionServer","multi.mutations":1} 2022-03-07 02:03:30,186 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646618310167","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300019,"client":"172.30.41.155:32920","multi.service_calls":0,"queuetimems":20586,"class":"HRegionServer","multi.mutations":1} 2022-03-07 02:03:34,113 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1206ms No GCs detected 2022-03-07 02:03:34,137 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646618313266","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300870,"client":"172.31.0.219:51938","multi.service_calls":0,"queuetimems":23684,"class":"HRegionServer","multi.mutations":2} 2022-03-07 02:03:44,687 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-07 02:05:47,669 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-07 02:05:47,991 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 337ms. 2022-03-07 02:05:49,383 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 2 failover attempts. Trying to failover after sleeping for 1420ms. 2022-03-07 02:05:52,465 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1578ms No GCs detected 2022-03-07 02:05:58,864 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2897ms No GCs detected 2022-03-07 02:05:58,999 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: tNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 3 failover attempts. Trying to failover after sleeping for 2424ms. 2022-03-07 02:06:32,231 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-07 02:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=225366, hits=217267, hitRatio=96.41%, , cachingAccesses=217386, cachingHits=216958, cachingHitsRatio=99.80%, evictions=16705, evicted=0, evictedPerRun=0.0 2022-03-07 02:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 02:07:15,605 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646617335603","responsesize":"26449","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 16 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1500002,"client":"172.31.0.219:51840","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":16} 2022-03-07 02:07:15,607 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646618235606","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:53882","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":5} 2022-03-07 02:07:15,646 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646618535645","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:53944","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 02:07:34,112 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646618554111","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:53944","multi.service_calls":0,"queuetimems":18466,"class":"HRegionServer","multi.mutations":1} 2022-03-07 02:07:45,404 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646618565404","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:53944","multi.service_calls":0,"queuetimems":29759,"class":"HRegionServer","multi.mutations":1} 2022-03-07 02:07:59,687 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646617979685","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 4 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:32904","multi.service_calls":0,"queuetimems":25576,"class":"HRegionServer","multi.mutations":4} 2022-03-07 02:08:03,276 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646617983275","responsesize":"14879","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 9 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:53822","multi.service_calls":0,"queuetimems":29165,"class":"HRegionServer","multi.mutations":9} 2022-03-07 02:08:12,396 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646618290808","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":601588,"client":"172.31.0.219:51938","multi.service_calls":0,"queuetimems":55202,"class":"HRegionServer","multi.mutations":2} 2022-03-07 02:08:12,753 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646618592752","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:53958","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 02:08:18,705 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646618598704","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300001,"client":"172.30.41.118:53960","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 02:08:30,187 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646618610186","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:32936","multi.service_calls":0,"queuetimems":34631,"class":"HRegionServer","multi.mutations":2} 2022-03-07 02:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=226162, hits=218063, hitRatio=96.42%, , cachingAccesses=218182, cachingHits=217754, cachingHitsRatio=99.80%, evictions=16735, evicted=0, evictedPerRun=0.0 2022-03-07 02:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 02:12:15,607 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646618835607","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:52000","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 02:12:15,608 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646618835607","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:52000","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 02:12:15,647 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646618835646","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:52000","multi.service_calls":0,"queuetimems":39,"class":"HRegionServer","multi.mutations":1} 2022-03-07 02:12:34,113 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646618854112","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:54008","multi.service_calls":0,"queuetimems":18504,"class":"HRegionServer","multi.mutations":1} 2022-03-07 02:12:45,405 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646618865404","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:54008","multi.service_calls":0,"queuetimems":29796,"class":"HRegionServer","multi.mutations":1} 2022-03-07 02:12:45,670 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646618265669","responsesize":"19835","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 12 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.31.0.219:51938","multi.service_calls":0,"queuetimems":30062,"class":"HRegionServer","multi.mutations":12} 2022-03-07 02:12:55,556 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646618575555","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 5 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:53954","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":5} 2022-03-07 02:12:59,687 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646618879687","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:54008","multi.service_calls":0,"queuetimems":44079,"class":"HRegionServer","multi.mutations":1} 2022-03-07 02:13:03,277 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646618883276","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:54008","multi.service_calls":0,"queuetimems":47668,"class":"HRegionServer","multi.mutations":1} 2022-03-07 02:13:12,397 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646618892396","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:54008","multi.service_calls":0,"queuetimems":56788,"class":"HRegionServer","multi.mutations":1} 2022-03-07 02:13:12,431 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646618289581","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 6 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":902850,"client":"172.30.41.155:32920","multi.service_calls":0,"queuetimems":53975,"class":"HRegionServer","multi.mutations":6} 2022-03-07 02:13:12,753 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646618892753","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:54008","multi.service_calls":0,"queuetimems":57146,"class":"HRegionServer","multi.mutations":1} 2022-03-07 02:13:18,706 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646618898705","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:54008","multi.service_calls":0,"queuetimems":63098,"class":"HRegionServer","multi.mutations":1} 2022-03-07 02:13:30,187 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646618910187","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:54008","multi.service_calls":0,"queuetimems":74579,"class":"HRegionServer","multi.mutations":1} 2022-03-07 02:13:32,410 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646618311902","responsesize":"18183","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 11 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900508,"client":"172.30.41.118:53882","multi.service_calls":0,"queuetimems":22321,"class":"HRegionServer","multi.mutations":11} 2022-03-07 02:13:34,145 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646618614144","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:53954","multi.service_calls":0,"queuetimems":3957,"class":"HRegionServer","multi.mutations":2} 2022-03-07 02:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=226997, hits=218898, hitRatio=96.43%, , cachingAccesses=219017, cachingHits=218589, cachingHitsRatio=99.80%, evictions=16765, evicted=0, evictedPerRun=0.0 2022-03-07 02:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 02:17:16,059 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646619135608","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300451,"client":"172.30.41.118:54056","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 02:17:16,061 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646619135608","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300453,"client":"172.30.41.118:54056","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 02:17:45,406 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646619165405","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:32970","multi.service_calls":0,"queuetimems":29757,"class":"HRegionServer","multi.mutations":1} 2022-03-07 02:17:45,671 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646619165670","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:32970","multi.service_calls":0,"queuetimems":30062,"class":"HRegionServer","multi.mutations":1} 2022-03-07 02:18:03,278 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646619183278","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:54066","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 02:18:12,462 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646619192461","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300001,"client":"172.30.41.118:54064","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 02:18:30,188 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646619210187","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:54066","multi.service_calls":0,"queuetimems":17433,"class":"HRegionServer","multi.mutations":1} 2022-03-07 02:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=227809, hits=219710, hitRatio=96.44%, , cachingAccesses=219829, cachingHits=219401, cachingHitsRatio=99.81%, evictions=16795, evicted=0, evictedPerRun=0.0 2022-03-07 02:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 02:22:16,061 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646619135648","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600412,"client":"172.30.41.118:54056","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 02:22:16,062 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646619436062","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:54124","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-07 02:22:16,063 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646619436062","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:54124","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-07 02:22:45,407 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646619465407","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:54124","multi.service_calls":0,"queuetimems":29345,"class":"HRegionServer","multi.mutations":1} 2022-03-07 02:22:45,671 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646619465671","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:54124","multi.service_calls":0,"queuetimems":29610,"class":"HRegionServer","multi.mutations":1} 2022-03-07 02:22:59,688 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646619179687","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:32970","multi.service_calls":0,"queuetimems":44039,"class":"HRegionServer","multi.mutations":4} 2022-03-07 02:23:03,279 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646619483278","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:54124","multi.service_calls":0,"queuetimems":47216,"class":"HRegionServer","multi.mutations":1} 2022-03-07 02:23:12,463 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646619492462","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:54124","multi.service_calls":0,"queuetimems":56400,"class":"HRegionServer","multi.mutations":1} 2022-03-07 02:23:12,754 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646619192753","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:52032","multi.service_calls":0,"queuetimems":9475,"class":"HRegionServer","multi.mutations":3} 2022-03-07 02:23:18,707 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646619198706","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:52032","multi.service_calls":0,"queuetimems":15428,"class":"HRegionServer","multi.mutations":2} 2022-03-07 02:23:30,189 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646619510189","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:54132","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-07 02:23:32,411 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646619212410","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 9 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:54066","multi.service_calls":0,"queuetimems":19656,"class":"HRegionServer","multi.mutations":9} 2022-03-07 02:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=228576, hits=220477, hitRatio=96.46%, , cachingAccesses=220596, cachingHits=220168, cachingHitsRatio=99.81%, evictions=16825, evicted=0, evictedPerRun=0.0 2022-03-07 02:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 02:27:16,063 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646619736062","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:54186","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 02:27:34,116 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646619154113","responsesize":"21487","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 13 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900002,"client":"172.30.41.118:54056","multi.service_calls":0,"queuetimems":18465,"class":"HRegionServer","multi.mutations":13} 2022-03-07 02:27:55,558 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646619175556","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 6 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.155:32970","multi.service_calls":0,"queuetimems":39947,"class":"HRegionServer","multi.mutations":6} 2022-03-07 02:28:12,398 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646619192397","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 8 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:54066","multi.service_calls":0,"queuetimems":9119,"class":"HRegionServer","multi.mutations":8} 2022-03-07 02:28:12,755 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646619792755","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:33008","multi.service_calls":0,"queuetimems":13066,"class":"HRegionServer","multi.mutations":1} 2022-03-07 02:28:30,190 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646619810189","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:54190","multi.service_calls":0,"queuetimems":17434,"class":"HRegionServer","multi.mutations":1} 2022-03-07 02:28:32,411 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646619812411","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:54190","multi.service_calls":0,"queuetimems":19656,"class":"HRegionServer","multi.mutations":1} 2022-03-07 02:28:34,147 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646619214146","responsesize":"16531","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 10 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.31.0.219:52032","multi.service_calls":0,"queuetimems":21391,"class":"HRegionServer","multi.mutations":10} 2022-03-07 02:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=229309, hits=221210, hitRatio=96.47%, , cachingAccesses=221329, cachingHits=220901, cachingHitsRatio=99.81%, evictions=16855, evicted=0, evictedPerRun=0.0 2022-03-07 02:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-07 02:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 02:32:34,118 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646620054117","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300001,"client":"172.30.41.118:54252","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-07 02:32:42,697 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-07 02:32:45,672 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646619765671","responsesize":"16528","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 10 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":600001,"client":"172.31.0.219:52096","multi.service_calls":0,"queuetimems":29609,"class":"HRegionServer","multi.mutations":10} 2022-03-07 02:32:59,690 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":1,"starttimems":"1646619779689","responsesize":"14884","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 10 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":600001,"client":"172.30.41.118:54190","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":9} 2022-03-07 02:33:12,399 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646620092399","responsesize":"13221","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 8 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:52116","multi.service_calls":0,"queuetimems":16840,"class":"HRegionServer","multi.mutations":8} 2022-03-07 02:33:12,464 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646619792463","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:52100","multi.service_calls":0,"queuetimems":12774,"class":"HRegionServer","multi.mutations":3} 2022-03-07 02:33:12,756 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646620092756","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:52116","multi.service_calls":0,"queuetimems":17197,"class":"HRegionServer","multi.mutations":1} 2022-03-07 02:33:30,191 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646620110190","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:54262","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 02:33:32,412 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646620112412","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:54258","multi.service_calls":0,"queuetimems":19625,"class":"HRegionServer","multi.mutations":2} 2022-03-07 02:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=230151, hits=222052, hitRatio=96.48%, , cachingAccesses=222171, cachingHits=221743, cachingHitsRatio=99.81%, evictions=16885, evicted=0, evictedPerRun=0.0 2022-03-07 02:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 02:37:16,064 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646619736063","responsesize":"14879","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 9 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:54186","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":9} 2022-03-07 02:37:16,064 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646619736063","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:52096","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":4} 2022-03-07 02:37:16,065 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646620036064","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:52116","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 02:37:34,120 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646620354119","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:54306","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 02:37:45,409 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646619765407","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 5 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.155:33006","multi.service_calls":0,"queuetimems":29344,"class":"HRegionServer","multi.mutations":5} 2022-03-07 02:37:59,691 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646620379691","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:33044","multi.service_calls":0,"queuetimems":25572,"class":"HRegionServer","multi.mutations":1} 2022-03-07 02:38:03,280 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646619783279","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:33008","multi.service_calls":0,"queuetimems":3590,"class":"HRegionServer","multi.mutations":6} 2022-03-07 02:38:12,466 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646620392464","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.118:54306","multi.service_calls":0,"queuetimems":26791,"class":"HRegionServer","multi.mutations":1} 2022-03-07 02:38:30,192 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646620410191","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:33044","multi.service_calls":0,"queuetimems":44518,"class":"HRegionServer","multi.mutations":1} 2022-03-07 02:38:34,162 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646620114148","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 6 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600014,"client":"172.30.41.118:54262","multi.service_calls":0,"queuetimems":3958,"class":"HRegionServer","multi.mutations":6} 2022-03-07 02:39:25,612 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3038ms No GCs detected 2022-03-07 02:39:58,808 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1900ms No GCs detected 2022-03-07 02:40:01,958 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1650ms No GCs detected 2022-03-07 02:40:15,255 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-07 02:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=230937, hits=222838, hitRatio=96.49%, , cachingAccesses=222957, cachingHits=222529, cachingHitsRatio=99.81%, evictions=16915, evicted=0, evictedPerRun=0.0 2022-03-07 02:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 02:42:16,066 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646620636065","responsesize":"6613","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:33062","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-07 02:42:34,120 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646620654120","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:54374","multi.service_calls":0,"queuetimems":18054,"class":"HRegionServer","multi.mutations":2} 2022-03-07 02:42:45,674 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646620365673","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 9 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:54306","multi.service_calls":0,"queuetimems":11554,"class":"HRegionServer","multi.mutations":9} 2022-03-07 02:42:55,564 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646620075558","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 6 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900006,"client":"172.31.0.219:52116","multi.service_calls":0,"queuetimems":39494,"class":"HRegionServer","multi.mutations":6} 2022-03-07 02:43:12,466 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646620692466","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.31.0.219:52158","multi.service_calls":0,"queuetimems":12743,"class":"HRegionServer","multi.mutations":2} 2022-03-07 02:43:12,757 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646620392756","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:52134","multi.service_calls":0,"queuetimems":38636,"class":"HRegionServer","multi.mutations":4} 2022-03-07 02:43:18,709 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646619798707","responsesize":"19838","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 12 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.31.0.219:52100","multi.service_calls":0,"queuetimems":19017,"class":"HRegionServer","multi.mutations":12} 2022-03-07 02:43:30,193 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646620710192","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:54380","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 02:43:34,163 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646620714162","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:33066","multi.service_calls":0,"queuetimems":3969,"class":"HRegionServer","multi.mutations":2} 2022-03-07 02:43:45,791 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-07 02:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=231672, hits=223573, hitRatio=96.50%, , cachingAccesses=223692, cachingHits=223264, cachingHitsRatio=99.81%, evictions=16945, evicted=0, evictedPerRun=0.0 2022-03-07 02:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 02:47:16,067 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646620636066","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:33062","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-07 02:47:37,792 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3009ms No GCs detected 2022-03-07 02:48:00,096 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646620679692","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600404,"client":"172.30.41.118:54376","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 02:48:01,955 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1980ms No GCs detected 2022-03-07 02:48:12,855 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646620392399","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900455,"client":"172.30.41.155:33044","multi.service_calls":0,"queuetimems":38279,"class":"HRegionServer","multi.mutations":7} 2022-03-07 02:48:12,855 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646620992466","responsesize":"6613","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300389,"client":"172.30.41.155:33080","multi.service_calls":0,"queuetimems":56399,"class":"HRegionServer","multi.mutations":4} 2022-03-07 02:48:16,243 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2512ms No GCs detected 2022-03-07 02:48:19,545 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-07 02:48:32,414 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646620412413","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 8 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:54306","multi.service_calls":0,"queuetimems":46740,"class":"HRegionServer","multi.mutations":8} 2022-03-07 02:48:35,908 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4198ms No GCs detected 2022-03-07 02:48:49,953 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-07 02:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=232465, hits=224366, hitRatio=96.52%, , cachingAccesses=224485, cachingHits=224057, cachingHitsRatio=99.81%, evictions=16975, evicted=0, evictedPerRun=0.0 2022-03-07 02:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 02:52:16,067 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646620636066","responsesize":"16531","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 10 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.30.41.118:54374","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":10} 2022-03-07 02:52:16,068 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646620936067","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 6 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:54410","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":6} 2022-03-07 02:52:45,410 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646620665409","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 6 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:52154","multi.service_calls":0,"queuetimems":29343,"class":"HRegionServer","multi.mutations":6} 2022-03-07 02:52:45,676 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646620965675","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:33080","multi.service_calls":0,"queuetimems":29608,"class":"HRegionServer","multi.mutations":3} 2022-03-07 02:53:03,282 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646620683281","responsesize":"18183","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 11 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.30.41.118:54376","multi.service_calls":0,"queuetimems":3589,"class":"HRegionServer","multi.mutations":11} 2022-03-07 02:53:12,855 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646621292855","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:54456","multi.service_calls":0,"queuetimems":56787,"class":"HRegionServer","multi.mutations":1} 2022-03-07 02:53:13,628 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646620992757","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600871,"client":"172.30.41.155:33080","multi.service_calls":0,"queuetimems":27082,"class":"HRegionServer","multi.mutations":2} 2022-03-07 02:53:19,542 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646620998709","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 6 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600833,"client":"172.30.41.118:54410","multi.service_calls":0,"queuetimems":33034,"class":"HRegionServer","multi.mutations":6} 2022-03-07 02:53:32,414 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646621312414","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:52228","multi.service_calls":0,"queuetimems":76346,"class":"HRegionServer","multi.mutations":1} 2022-03-07 02:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=233199, hits=225100, hitRatio=96.53%, , cachingAccesses=225219, cachingHits=224791, cachingHitsRatio=99.81%, evictions=17005, evicted=0, evictedPerRun=0.0 2022-03-07 02:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 02:56:51,306 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-07 02:57:16,069 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646621236068","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 5 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:33098","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":5} 2022-03-07 02:57:39,532 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646620954120","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":905412,"client":"172.30.41.118:54410","multi.service_calls":0,"queuetimems":18053,"class":"HRegionServer","multi.mutations":6} 2022-03-07 02:57:55,566 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646620975564","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.31.0.219:52184","multi.service_calls":0,"queuetimems":39496,"class":"HRegionServer","multi.mutations":7} 2022-03-07 02:58:00,097 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646621280096","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:54456","multi.service_calls":0,"queuetimems":44028,"class":"HRegionServer","multi.mutations":2} 2022-03-07 02:58:03,283 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646621583283","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:54500","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 02:58:12,856 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646621292855","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:54456","multi.service_calls":0,"queuetimems":56787,"class":"HRegionServer","multi.mutations":6} 2022-03-07 02:58:19,542 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646621599542","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:54500","multi.service_calls":0,"queuetimems":5913,"class":"HRegionServer","multi.mutations":1} 2022-03-07 02:58:33,108 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3855ms No GCs detected 2022-03-07 02:58:36,487 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2879ms No GCs detected 2022-03-07 02:58:36,518 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646621014163","responsesize":"19835","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 12 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":902355,"client":"172.30.41.118:54424","multi.service_calls":0,"queuetimems":3969,"class":"HRegionServer","multi.mutations":12} 2022-03-07 02:58:45,564 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1550ms No GCs detected 2022-03-07 03:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=234017, hits=225918, hitRatio=96.54%, , cachingAccesses=226037, cachingHits=225609, cachingHitsRatio=99.81%, evictions=17035, evicted=0, evictedPerRun=0.0 2022-03-07 03:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 03:02:16,069 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646621536069","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:54494","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 03:02:16,070 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646621536069","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.31.0.219:52252","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 03:02:16,070 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646621836070","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:54536","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 03:02:45,411 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646621565410","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:54494","multi.service_calls":0,"queuetimems":29341,"class":"HRegionServer","multi.mutations":4} 2022-03-07 03:02:55,567 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646621875566","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:54536","multi.service_calls":0,"queuetimems":39496,"class":"HRegionServer","multi.mutations":2} 2022-03-07 03:03:00,098 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646621880097","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.30.41.118:54370","multi.service_calls":0,"queuetimems":25892,"class":"HRegionServer","multi.mutations":2} 2022-03-07 03:03:03,284 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646621883283","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:52272","multi.service_calls":0,"queuetimems":22924,"class":"HRegionServer","multi.mutations":1} 2022-03-07 03:03:12,856 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646621892856","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:54536","multi.service_calls":0,"queuetimems":30430,"class":"HRegionServer","multi.mutations":1} 2022-03-07 03:03:14,784 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646621592855","responsesize":"16528","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 10 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":601929,"client":"172.30.41.118:54500","multi.service_calls":0,"queuetimems":9572,"class":"HRegionServer","multi.mutations":10} 2022-03-07 03:03:14,785 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646621593629","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":601156,"client":"172.30.41.155:33132","multi.service_calls":0,"queuetimems":10346,"class":"HRegionServer","multi.mutations":2} 2022-03-07 03:03:14,784 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2920ms No GCs detected 2022-03-07 03:03:19,543 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646621899543","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:54536","multi.service_calls":0,"queuetimems":41870,"class":"HRegionServer","multi.mutations":1} 2022-03-07 03:03:23,836 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-07 03:03:30,738 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646621010194","responsesize":"14882","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 9 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200544,"client":"172.30.41.155:33082","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":9} 2022-03-07 03:03:36,519 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646621916518","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300001,"client":"172.30.41.118:54536","multi.service_calls":0,"queuetimems":56985,"class":"HRegionServer","multi.mutations":1} 2022-03-07 03:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=234868, hits=226769, hitRatio=96.55%, , cachingAccesses=226888, cachingHits=226460, cachingHitsRatio=99.81%, evictions=17065, evicted=0, evictedPerRun=0.0 2022-03-07 03:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 03:07:16,071 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646622136071","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:54592","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 03:07:39,534 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646621859532","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.31.0.219:52272","multi.service_calls":0,"queuetimems":23462,"class":"HRegionServer","multi.mutations":2} 2022-03-07 03:07:45,677 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646621565676","responsesize":"16531","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 10 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.31.0.219:52252","multi.service_calls":0,"queuetimems":29607,"class":"HRegionServer","multi.mutations":10} 2022-03-07 03:07:55,568 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646622175568","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:54596","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 03:08:03,285 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646622183284","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:54598","multi.service_calls":0,"queuetimems":7686,"class":"HRegionServer","multi.mutations":2} 2022-03-07 03:08:12,857 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646622192857","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:33206","multi.service_calls":0,"queuetimems":12759,"class":"HRegionServer","multi.mutations":1} 2022-03-07 03:08:16,582 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646622196579","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300003,"client":"172.30.41.155:33206","multi.service_calls":0,"queuetimems":21011,"class":"HRegionServer","multi.mutations":1} 2022-03-07 03:08:16,583 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646622196579","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300004,"client":"172.30.41.155:33206","multi.service_calls":0,"queuetimems":16480,"class":"HRegionServer","multi.mutations":3} 2022-03-07 03:08:32,584 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646622212583","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:52298","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 03:08:36,530 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646621612415","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 7 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":904115,"client":"172.30.41.118:54500","multi.service_calls":0,"queuetimems":18786,"class":"HRegionServer","multi.mutations":7} 2022-03-07 03:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=235663, hits=227564, hitRatio=96.56%, , cachingAccesses=227683, cachingHits=227255, cachingHitsRatio=99.81%, evictions=17095, evicted=0, evictedPerRun=0.0 2022-03-07 03:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 03:12:39,535 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646622459535","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:52312","multi.service_calls":0,"queuetimems":23463,"class":"HRegionServer","multi.mutations":1} 2022-03-07 03:12:55,569 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646622475568","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:54646","multi.service_calls":0,"queuetimems":9890,"class":"HRegionServer","multi.mutations":1} 2022-03-07 03:13:03,285 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646622483285","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:54646","multi.service_calls":0,"queuetimems":47213,"class":"HRegionServer","multi.mutations":3} 2022-03-07 03:13:16,583 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646622496582","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.31.0.219:52254","multi.service_calls":0,"queuetimems":12260,"class":"HRegionServer","multi.mutations":2} 2022-03-07 03:13:32,584 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646622512584","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:33226","multi.service_calls":0,"queuetimems":28144,"class":"HRegionServer","multi.mutations":1} 2022-03-07 03:13:36,531 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646622516530","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:33226","multi.service_calls":0,"queuetimems":32086,"class":"HRegionServer","multi.mutations":1} 2022-03-07 03:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=236542, hits=228443, hitRatio=96.58%, , cachingAccesses=228562, cachingHits=228134, cachingHitsRatio=99.81%, evictions=17125, evicted=0, evictedPerRun=0.0 2022-03-07 03:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 03:17:16,072 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646622136071","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:33202","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 03:17:16,073 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646622436072","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:33226","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 03:17:26,276 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-07 03:17:38,651 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1394ms No GCs detected 2022-03-07 03:17:39,538 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646622759536","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.118:54706","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 03:17:45,969 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646622465678","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600291,"client":"172.30.41.118:54646","multi.service_calls":0,"queuetimems":29606,"class":"HRegionServer","multi.mutations":4} 2022-03-07 03:17:55,571 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646622775570","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:54706","multi.service_calls":0,"queuetimems":16034,"class":"HRegionServer","multi.mutations":1} 2022-03-07 03:18:03,286 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646622783285","responsesize":"6613","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:33244","multi.service_calls":0,"queuetimems":23749,"class":"HRegionServer","multi.mutations":4} 2022-03-07 03:18:16,583 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646622796583","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:54706","multi.service_calls":0,"queuetimems":37047,"class":"HRegionServer","multi.mutations":1} 2022-03-07 03:18:16,584 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646622496583","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:54646","multi.service_calls":0,"queuetimems":30905,"class":"HRegionServer","multi.mutations":4} 2022-03-07 03:18:19,546 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646622199543","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.155:33206","multi.service_calls":0,"queuetimems":23975,"class":"HRegionServer","multi.mutations":7} 2022-03-07 03:18:27,115 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-07 03:18:36,521 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646622216519","responsesize":"19835","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 12 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900002,"client":"172.31.0.219:52298","multi.service_calls":0,"queuetimems":3936,"class":"HRegionServer","multi.mutations":12} 2022-03-07 03:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=237303, hits=229204, hitRatio=96.59%, , cachingAccesses=229323, cachingHits=228895, cachingHitsRatio=99.81%, evictions=17155, evicted=0, evictedPerRun=0.0 2022-03-07 03:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 03:22:16,073 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646622136071","responsesize":"14882","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 9 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200002,"client":"172.30.41.155:33202","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":9} 2022-03-07 03:22:16,074 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646623036073","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300000,"client":"172.30.41.118:54764","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 03:22:39,538 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646623059538","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:33260","multi.service_calls":0,"queuetimems":23465,"class":"HRegionServer","multi.mutations":1} 2022-03-07 03:23:00,334 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646622180098","responsesize":"21490","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 13 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200236,"client":"172.31.0.219:52296","multi.service_calls":0,"queuetimems":4530,"class":"HRegionServer","multi.mutations":13} 2022-03-07 03:23:03,286 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646623083286","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:54764","multi.service_calls":0,"queuetimems":47212,"class":"HRegionServer","multi.mutations":1} 2022-03-07 03:23:16,584 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646623096584","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:33260","multi.service_calls":0,"queuetimems":60510,"class":"HRegionServer","multi.mutations":1} 2022-03-07 03:23:19,548 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646623099546","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.118:54772","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 03:23:29,993 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-07 03:23:36,522 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646623116521","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300000,"client":"172.30.41.118:54772","multi.service_calls":0,"queuetimems":16975,"class":"HRegionServer","multi.mutations":1} 2022-03-07 03:25:16,179 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1385ms No GCs detected 2022-03-07 03:25:22,324 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 5644ms No GCs detected 2022-03-07 03:25:30,890 INFO [regionserver/cvp328:16201.Chore.3] hbase.ScheduledChore: Chore: CompactionChecker missed its start time 2022-03-07 03:25:30,890 INFO [regionserver/cvp328:16201.Chore.2] hbase.ScheduledChore: Chore: MemstoreFlusherChore missed its start time 2022-03-07 03:25:30,871 WARN [regionserver/cvp328:16201] util.Sleeper: We slept 20488ms instead of 5000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired 2022-03-07 03:26:13,663 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2845ms No GCs detected 2022-03-07 03:26:32,675 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-07 03:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=562.40 KB, freeSize=306.65 MB, max=307.20 MB, blockCount=23, accesses=238139, hits=230040, hitRatio=96.60%, , cachingAccesses=230159, cachingHits=229731, cachingHitsRatio=99.81%, evictions=17183, evicted=0, evictedPerRun=0.0 2022-03-07 03:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 03:27:16,075 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646623336074","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:54824","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 03:27:16,075 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646623336074","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:33274","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 03:27:39,540 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646623359539","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:54824","multi.service_calls":0,"queuetimems":23465,"class":"HRegionServer","multi.mutations":3} 2022-03-07 03:27:45,414 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646622165412","responsesize":"29753","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 18 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1500002,"client":"172.30.41.118:54592","multi.service_calls":0,"queuetimems":29341,"class":"HRegionServer","multi.mutations":18} 2022-03-07 03:27:45,970 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646623065969","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:33260","multi.service_calls":0,"queuetimems":29896,"class":"HRegionServer","multi.mutations":2} 2022-03-07 03:27:55,572 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646623075571","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:52376","multi.service_calls":0,"queuetimems":39497,"class":"HRegionServer","multi.mutations":2} 2022-03-07 03:28:00,335 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646623380334","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:52410","multi.service_calls":0,"queuetimems":20795,"class":"HRegionServer","multi.mutations":1} 2022-03-07 03:28:12,859 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646622492857","responsesize":"21490","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 13 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.31.0.219:52312","multi.service_calls":0,"queuetimems":56784,"class":"HRegionServer","multi.mutations":13} 2022-03-07 03:28:16,584 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646623096583","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 6 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:33260","multi.service_calls":0,"queuetimems":60509,"class":"HRegionServer","multi.mutations":6} 2022-03-07 03:28:32,587 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646622812585","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:33246","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":6} 2022-03-07 03:28:36,523 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646623416523","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:54834","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 03:28:36,532 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646622816531","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 7 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:52350","multi.service_calls":0,"queuetimems":3945,"class":"HRegionServer","multi.mutations":7} 2022-03-07 03:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=577.19 KB, freeSize=306.64 MB, max=307.20 MB, blockCount=24, accesses=238843, hits=230743, hitRatio=96.61%, , cachingAccesses=230863, cachingHits=230434, cachingHitsRatio=99.81%, evictions=17213, evicted=0, evictedPerRun=0.0 2022-03-07 03:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-07 03:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 03:32:16,075 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646623036073","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 8 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.118:54764","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":8} 2022-03-07 03:32:16,076 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646623636076","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:33288","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 03:32:16,076 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646623636076","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:54866","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 03:32:39,542 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646623659540","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.155:33288","multi.service_calls":0,"queuetimems":23464,"class":"HRegionServer","multi.mutations":1} 2022-03-07 03:32:45,417 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646623665416","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:52448","multi.service_calls":0,"queuetimems":29340,"class":"HRegionServer","multi.mutations":3} 2022-03-07 03:32:45,971 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646623665970","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:52448","multi.service_calls":0,"queuetimems":29894,"class":"HRegionServer","multi.mutations":1} 2022-03-07 03:33:00,335 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646623680335","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:33288","multi.service_calls":0,"queuetimems":44259,"class":"HRegionServer","multi.mutations":1} 2022-03-07 03:33:03,287 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646623383286","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 8 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:54824","multi.service_calls":0,"queuetimems":23747,"class":"HRegionServer","multi.mutations":8} 2022-03-07 03:33:12,860 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646623692860","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:33288","multi.service_calls":0,"queuetimems":56784,"class":"HRegionServer","multi.mutations":1} 2022-03-07 03:33:16,585 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646623396584","responsesize":"16528","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 10 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":600001,"client":"172.31.0.219:52410","multi.service_calls":0,"queuetimems":37044,"class":"HRegionServer","multi.mutations":10} 2022-03-07 03:33:19,549 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646623399548","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:33274","multi.service_calls":0,"queuetimems":63474,"class":"HRegionServer","multi.mutations":5} 2022-03-07 03:33:36,524 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646623716524","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:54880","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 03:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=650.08 KB, freeSize=306.57 MB, max=307.20 MB, blockCount=26, accesses=239607, hits=231505, hitRatio=96.62%, , cachingAccesses=231627, cachingHits=231196, cachingHitsRatio=99.81%, evictions=17243, evicted=0, evictedPerRun=0.0 2022-03-07 03:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 03:37:16,077 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646623936077","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:54914","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 03:37:16,078 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":1,"starttimems":"1646623936076","responsesize":"1665","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x95\\xC4\\x04tags ","processingtimems":300002,"client":"172.30.41.155:33300","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 03:37:45,418 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646623965417","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:54914","multi.service_calls":0,"queuetimems":29341,"class":"HRegionServer","multi.mutations":1} 2022-03-07 03:37:55,573 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646623675572","responsesize":"16528","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 10 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":600001,"client":"172.30.41.118:54866","multi.service_calls":0,"queuetimems":39496,"class":"HRegionServer","multi.mutations":10} 2022-03-07 03:38:00,336 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646623980336","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:33300","multi.service_calls":0,"queuetimems":44259,"class":"HRegionServer","multi.mutations":1} 2022-03-07 03:38:16,585 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646623696584","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:52448","multi.service_calls":0,"queuetimems":37043,"class":"HRegionServer","multi.mutations":3} 2022-03-07 03:38:16,585 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646623996585","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.31.0.219:52490","multi.service_calls":0,"queuetimems":13266,"class":"HRegionServer","multi.mutations":2} 2022-03-07 03:38:19,551 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646623999549","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.31.0.219:52488","multi.service_calls":0,"queuetimems":16260,"class":"HRegionServer","multi.mutations":1} 2022-03-07 03:38:36,525 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646624016525","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:52488","multi.service_calls":0,"queuetimems":16975,"class":"HRegionServer","multi.mutations":1} 2022-03-07 03:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=650.08 KB, freeSize=306.57 MB, max=307.20 MB, blockCount=26, accesses=240431, hits=232329, hitRatio=96.63%, , cachingAccesses=232451, cachingHits=232020, cachingHitsRatio=99.81%, evictions=17273, evicted=0, evictedPerRun=0.0 2022-03-07 03:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 03:42:16,079 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646624236078","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:54968","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 03:42:39,545 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":4,"starttimems":"1646623959542","responsesize":"11073","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 9 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x95\\xC4\\x04tags ","processingtimems":600002,"client":"172.30.41.155:33300","multi.service_calls":0,"queuetimems":23466,"class":"HRegionServer","multi.mutations":5} 2022-03-07 03:43:00,338 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646624280337","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:52518","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 03:43:12,861 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646623992860","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 5 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:52488","multi.service_calls":0,"queuetimems":9571,"class":"HRegionServer","multi.mutations":5} 2022-03-07 03:43:16,586 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646624296586","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.31.0.219:52520","multi.service_calls":0,"queuetimems":16218,"class":"HRegionServer","multi.mutations":2} 2022-03-07 03:43:19,552 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646624299552","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:52518","multi.service_calls":0,"queuetimems":2965,"class":"HRegionServer","multi.mutations":1} 2022-03-07 03:43:32,588 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646623712587","responsesize":"16531","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 10 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.31.0.219:52448","multi.service_calls":0,"queuetimems":53046,"class":"HRegionServer","multi.mutations":10} 2022-03-07 03:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=650.08 KB, freeSize=306.57 MB, max=307.20 MB, blockCount=26, accesses=241257, hits=233155, hitRatio=96.64%, , cachingAccesses=233277, cachingHits=232846, cachingHitsRatio=99.82%, evictions=17303, evicted=0, evictedPerRun=0.0 2022-03-07 03:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 03:47:16,078 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646623936076","responsesize":"19835","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 12 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900002,"client":"172.30.41.118:54914","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":12} 2022-03-07 03:47:16,080 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646624536080","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:52550","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 03:47:39,547 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646624559545","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.31.0.219:52550","multi.service_calls":0,"queuetimems":23465,"class":"HRegionServer","multi.mutations":2} 2022-03-07 03:47:45,419 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646624265418","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:33316","multi.service_calls":0,"queuetimems":29339,"class":"HRegionServer","multi.mutations":3} 2022-03-07 03:47:45,973 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646623965971","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 5 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.155:33300","multi.service_calls":0,"queuetimems":29894,"class":"HRegionServer","multi.mutations":5} 2022-03-07 03:48:00,339 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646624580338","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300000,"client":"172.30.41.118:55020","multi.service_calls":0,"queuetimems":22770,"class":"HRegionServer","multi.mutations":1} 2022-03-07 03:48:03,290 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646623983288","responsesize":"16531","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 10 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900002,"client":"172.30.41.118:54926","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":10} 2022-03-07 03:48:12,863 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646624592862","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:55032","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 03:48:16,586 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646624296585","responsesize":"18180","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 11 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":600001,"client":"172.31.0.219:52518","multi.service_calls":0,"queuetimems":16247,"class":"HRegionServer","multi.mutations":11} 2022-03-07 03:48:16,586 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646624596586","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:55032","multi.service_calls":0,"queuetimems":3724,"class":"HRegionServer","multi.mutations":2} 2022-03-07 03:48:32,589 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646624612588","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:52556","multi.service_calls":0,"queuetimems":19726,"class":"HRegionServer","multi.mutations":1} 2022-03-07 03:48:36,526 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646624316525","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:33318","multi.service_calls":0,"queuetimems":16973,"class":"HRegionServer","multi.mutations":3} 2022-03-07 03:48:36,534 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646623716532","responsesize":"18186","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 11 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.118:54880","multi.service_calls":0,"queuetimems":8,"class":"HRegionServer","multi.mutations":11} 2022-03-07 03:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=650.08 KB, freeSize=306.57 MB, max=307.20 MB, blockCount=26, accesses=242054, hits=233952, hitRatio=96.65%, , cachingAccesses=234074, cachingHits=233643, cachingHitsRatio=99.82%, evictions=17333, evicted=0, evictedPerRun=0.0 2022-03-07 03:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 03:52:55,576 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":5,"starttimems":"1646624275573","responsesize":"17693","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 14 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900003,"client":"172.30.41.155:33316","multi.service_calls":0,"queuetimems":39494,"class":"HRegionServer","multi.mutations":9} 2022-03-07 03:53:03,290 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646624883290","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:33356","multi.service_calls":0,"queuetimems":47209,"class":"HRegionServer","multi.mutations":1} 2022-03-07 03:53:16,587 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646624896587","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:55092","multi.service_calls":0,"queuetimems":3723,"class":"HRegionServer","multi.mutations":2} 2022-03-07 03:53:19,555 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646624599553","responsesize":"11572","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 7 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.31.0.219:52556","multi.service_calls":0,"queuetimems":6690,"class":"HRegionServer","multi.mutations":7} 2022-03-07 03:53:32,589 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646624912589","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300000,"client":"172.30.41.118:55092","multi.service_calls":0,"queuetimems":16002,"class":"HRegionServer","multi.mutations":1} 2022-03-07 03:53:36,527 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646624916526","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:52576","multi.service_calls":0,"queuetimems":19938,"class":"HRegionServer","multi.mutations":2} 2022-03-07 03:53:36,537 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646624916534","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300003,"client":"172.30.41.118:55094","multi.service_calls":0,"queuetimems":3914,"class":"HRegionServer","multi.mutations":2} 2022-03-07 03:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=650.08 KB, freeSize=306.57 MB, max=307.20 MB, blockCount=26, accesses=242810, hits=234708, hitRatio=96.66%, , cachingAccesses=234830, cachingHits=234399, cachingHitsRatio=99.82%, evictions=17363, evicted=0, evictedPerRun=0.0 2022-03-07 03:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 03:57:16,081 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646624836080","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:55084","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":2} 2022-03-07 03:57:16,082 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646624836081","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:33356","multi.service_calls":0,"queuetimems":2,"class":"HRegionServer","multi.mutations":3} 2022-03-07 03:57:39,548 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646624859548","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:55084","multi.service_calls":0,"queuetimems":23469,"class":"HRegionServer","multi.mutations":4} 2022-03-07 03:57:45,420 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646624865419","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.31.0.219:52574","multi.service_calls":0,"queuetimems":29339,"class":"HRegionServer","multi.mutations":3} 2022-03-07 03:57:55,577 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646625175577","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:33376","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 03:58:00,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646624880339","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 5 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:52574","multi.service_calls":0,"queuetimems":44259,"class":"HRegionServer","multi.mutations":5} 2022-03-07 03:58:32,590 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646625212589","responsesize":"6613","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:52592","multi.service_calls":0,"queuetimems":37012,"class":"HRegionServer","multi.mutations":4} 2022-03-07 04:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=650.08 KB, freeSize=306.57 MB, max=307.20 MB, blockCount=26, accesses=243635, hits=235533, hitRatio=96.67%, , cachingAccesses=235655, cachingHits=235224, cachingHitsRatio=99.82%, evictions=17393, evicted=0, evictedPerRun=0.0 2022-03-07 04:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 04:02:16,081 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646624236079","responsesize":"29753","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 18 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1500002,"client":"172.30.41.118:54968","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":18} 2022-03-07 04:02:39,550 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646625459549","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:33394","multi.service_calls":0,"queuetimems":23466,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:02:45,420 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646625465420","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:52610","multi.service_calls":0,"queuetimems":29337,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:02:45,975 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":3,"starttimems":"1646624865973","responsesize":"13251","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 11 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900002,"client":"172.30.41.155:33356","multi.service_calls":0,"queuetimems":29892,"class":"HRegionServer","multi.mutations":8} 2022-03-07 04:02:55,578 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646625475577","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:52610","multi.service_calls":0,"queuetimems":39495,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:03:00,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646625480340","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:52610","multi.service_calls":0,"queuetimems":44257,"class":"HRegionServer","multi.mutations":2} 2022-03-07 04:03:03,292 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646625183291","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:33376","multi.service_calls":0,"queuetimems":7714,"class":"HRegionServer","multi.mutations":2} 2022-03-07 04:03:12,866 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646624892864","responsesize":"16531","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 10 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900002,"client":"172.30.41.118:55092","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":10} 2022-03-07 04:03:16,588 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646624896587","responsesize":"21487","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 13 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.31.0.219:52576","multi.service_calls":0,"queuetimems":3722,"class":"HRegionServer","multi.mutations":13} 2022-03-07 04:03:19,556 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646625199555","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 5 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:52592","multi.service_calls":0,"queuetimems":23978,"class":"HRegionServer","multi.mutations":5} 2022-03-07 04:03:32,591 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646625512591","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:55212","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=650.08 KB, freeSize=306.57 MB, max=307.20 MB, blockCount=26, accesses=244354, hits=236252, hitRatio=96.68%, , cachingAccesses=236374, cachingHits=235943, cachingHitsRatio=99.82%, evictions=17423, evicted=0, evictedPerRun=0.0 2022-03-07 04:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 04:07:16,083 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646625436082","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":600001,"client":"172.30.41.118:55200","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 04:07:16,083 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646625436082","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:33394","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 04:07:45,421 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646625765420","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:52628","multi.service_calls":0,"queuetimems":29338,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:08:00,341 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646625780340","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:33412","multi.service_calls":0,"queuetimems":20789,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:08:03,292 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646625783292","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:33412","multi.service_calls":0,"queuetimems":23741,"class":"HRegionServer","multi.mutations":2} 2022-03-07 04:08:16,589 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646625796588","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:52628","multi.service_calls":0,"queuetimems":37036,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:08:36,529 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":3,"starttimems":"1646625216527","responsesize":"17412","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 11 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900002,"client":"172.30.41.155:33376","multi.service_calls":0,"queuetimems":33235,"class":"HRegionServer","multi.mutations":8} 2022-03-07 04:08:36,538 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646625216537","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 6 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:52592","multi.service_calls":0,"queuetimems":33245,"class":"HRegionServer","multi.mutations":6} 2022-03-07 04:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=650.08 KB, freeSize=306.57 MB, max=307.20 MB, blockCount=26, accesses=245114, hits=237012, hitRatio=96.69%, , cachingAccesses=237134, cachingHits=236703, cachingHitsRatio=99.82%, evictions=17453, evicted=0, evictedPerRun=0.0 2022-03-07 04:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 04:12:16,083 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646625736082","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:33412","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 04:12:16,084 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646626036084","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:55310","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:12:16,084 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929056007 2022-03-07 04:12:39,552 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646625759551","responsesize":"11572","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 7 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":600001,"client":"172.30.41.118:55254","multi.service_calls":0,"queuetimems":23469,"class":"HRegionServer","multi.mutations":7} 2022-03-07 04:12:45,976 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":3,"starttimems":"1646625765975","responsesize":"14104","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 9 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:33412","multi.service_calls":0,"queuetimems":29893,"class":"HRegionServer","multi.mutations":6} 2022-03-07 04:13:00,341 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646626080341","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:33432","multi.service_calls":0,"queuetimems":44257,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:13:03,293 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646626083293","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:33432","multi.service_calls":0,"queuetimems":47209,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:13:19,558 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646625799556","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.118:55254","multi.service_calls":0,"queuetimems":40005,"class":"HRegionServer","multi.mutations":2} 2022-03-07 04:13:36,530 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646626116529","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:33432","multi.service_calls":0,"queuetimems":80445,"class":"HRegionServer","multi.mutations":3} 2022-03-07 04:13:36,539 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646626116538","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:33432","multi.service_calls":0,"queuetimems":80454,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=650.08 KB, freeSize=306.57 MB, max=307.20 MB, blockCount=26, accesses=245971, hits=237869, hitRatio=96.71%, , cachingAccesses=237991, cachingHits=237560, cachingHitsRatio=99.82%, evictions=17483, evicted=0, evictedPerRun=0.0 2022-03-07 04:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 04:17:16,085 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646626036084","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:33432","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 04:17:16,085 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646626336085","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:55374","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 04:17:16,086 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929056419 2022-03-07 04:17:39,554 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646626359552","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.31.0.219:52664","multi.service_calls":0,"queuetimems":23467,"class":"HRegionServer","multi.mutations":3} 2022-03-07 04:17:45,977 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646626365976","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:55374","multi.service_calls":0,"queuetimems":29891,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:17:55,579 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646625775578","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 6 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:52628","multi.service_calls":0,"queuetimems":39495,"class":"HRegionServer","multi.mutations":6} 2022-03-07 04:18:00,343 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646626380342","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:55378","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:18:03,294 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646626383293","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:55378","multi.service_calls":0,"queuetimems":2951,"class":"HRegionServer","multi.mutations":2} 2022-03-07 04:18:12,867 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646625792866","responsesize":"16531","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 10 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.30.41.118:55254","multi.service_calls":0,"queuetimems":33315,"class":"HRegionServer","multi.mutations":10} 2022-03-07 04:18:16,590 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646625196587","responsesize":"29753","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 18 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1500003,"client":"172.30.41.118:55146","multi.service_calls":0,"queuetimems":21010,"class":"HRegionServer","multi.mutations":18} 2022-03-07 04:18:16,591 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646626096590","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:55322","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 04:18:36,539 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646626416539","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:33456","multi.service_calls":0,"queuetimems":16980,"class":"HRegionServer","multi.mutations":2} 2022-03-07 04:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=650.08 KB, freeSize=306.57 MB, max=307.20 MB, blockCount=26, accesses=246766, hits=238664, hitRatio=96.72%, , cachingAccesses=238786, cachingHits=238355, cachingHitsRatio=99.82%, evictions=17513, evicted=0, evictedPerRun=0.0 2022-03-07 04:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 04:22:16,171 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646626636086","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300085,"client":"172.30.41.155:33488","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:22:16,178 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646626336085","responsesize":"11572","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600092,"client":"172.30.41.155:33454","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":7} 2022-03-07 04:22:25,166 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3435ms No GCs detected 2022-03-07 04:22:36,171 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-07 04:22:45,977 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646626665977","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:55436","multi.service_calls":0,"queuetimems":29891,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:23:00,374 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646626680374","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.155:33492","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 04:23:05,533 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646626685533","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:33494","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:23:16,591 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646626696591","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:33494","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:23:16,591 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646626696591","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:33494","multi.service_calls":0,"queuetimems":11058,"class":"HRegionServer","multi.mutations":2} 2022-03-07 04:23:19,561 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646626399559","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 6 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.155:33456","multi.service_calls":0,"queuetimems":19217,"class":"HRegionServer","multi.mutations":6} 2022-03-07 04:23:32,594 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646625812592","responsesize":"11578","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 7 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200002,"client":"172.30.41.155:33414","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":7} 2022-03-07 04:23:36,531 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646626416530","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:52666","multi.service_calls":0,"queuetimems":36187,"class":"HRegionServer","multi.mutations":4} 2022-03-07 04:23:36,540 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646626716539","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:55440","multi.service_calls":0,"queuetimems":19948,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=650.08 KB, freeSize=306.57 MB, max=307.20 MB, blockCount=26, accesses=247469, hits=239367, hitRatio=96.73%, , cachingAccesses=239489, cachingHits=239058, cachingHitsRatio=99.82%, evictions=17543, evicted=0, evictedPerRun=0.0 2022-03-07 04:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 04:27:16,171 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646626636087","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600084,"client":"172.30.41.155:33488","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":2} 2022-03-07 04:27:16,174 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646626936174","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:52704","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:27:16,178 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646626936178","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:52704","multi.service_calls":0,"queuetimems":5,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:27:45,978 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646626965977","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:52704","multi.service_calls":0,"queuetimems":29804,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:28:00,375 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646626980375","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:55478","multi.service_calls":0,"queuetimems":44200,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:28:05,534 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646626985533","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:55478","multi.service_calls":0,"queuetimems":49358,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:28:12,868 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646626692867","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:33494","multi.service_calls":0,"queuetimems":7334,"class":"HRegionServer","multi.mutations":4} 2022-03-07 04:28:16,592 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646626996591","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:55478","multi.service_calls":0,"queuetimems":60416,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:28:16,592 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646626996592","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:55478","multi.service_calls":0,"queuetimems":60417,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:28:19,561 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646626999561","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300000,"client":"172.30.41.118:55478","multi.service_calls":0,"queuetimems":63386,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:28:32,595 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646627012594","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:55478","multi.service_calls":0,"queuetimems":76419,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:28:36,532 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646627016531","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:55478","multi.service_calls":0,"queuetimems":80356,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:28:36,609 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646627016540","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300069,"client":"172.30.41.118:55478","multi.service_calls":0,"queuetimems":80365,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=650.08 KB, freeSize=306.57 MB, max=307.20 MB, blockCount=26, accesses=248240, hits=240138, hitRatio=96.74%, , cachingAccesses=240260, cachingHits=239829, cachingHitsRatio=99.82%, evictions=17573, evicted=0, evictedPerRun=0.0 2022-03-07 04:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-07 04:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 04:32:16,173 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646627236172","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:55514","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:32:16,179 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646627236178","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:55514","multi.service_calls":0,"queuetimems":5,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:32:45,425 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646626065421","responsesize":"29753","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 18 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1500003,"client":"172.30.41.118:55310","multi.service_calls":0,"queuetimems":29337,"class":"HRegionServer","multi.mutations":18} 2022-03-07 04:33:00,376 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646627280375","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:33554","multi.service_calls":0,"queuetimems":44203,"class":"HRegionServer","multi.mutations":2} 2022-03-07 04:33:05,534 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646627285534","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:52716","multi.service_calls":0,"queuetimems":49361,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:33:12,869 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646627292868","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:55514","multi.service_calls":0,"queuetimems":56695,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:33:32,595 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646627312595","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300000,"client":"172.30.41.118:55522","multi.service_calls":0,"queuetimems":16002,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:33:36,610 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646627316609","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.30.41.155:33560","multi.service_calls":0,"queuetimems":19986,"class":"HRegionServer","multi.mutations":2} 2022-03-07 04:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=650.08 KB, freeSize=306.57 MB, max=307.20 MB, blockCount=26, accesses=249061, hits=240959, hitRatio=96.75%, , cachingAccesses=241081, cachingHits=240650, cachingHitsRatio=99.82%, evictions=17603, evicted=0, evictedPerRun=0.0 2022-03-07 04:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 04:37:16,175 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646627536174","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:55552","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:37:16,176 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646627236175","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:33554","multi.service_calls":0,"queuetimems":2,"class":"HRegionServer","multi.mutations":5} 2022-03-07 04:37:16,180 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646627536179","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:55552","multi.service_calls":0,"queuetimems":5,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:37:45,979 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646627265978","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 8 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:55514","multi.service_calls":0,"queuetimems":29805,"class":"HRegionServer","multi.mutations":8} 2022-03-07 04:37:55,581 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646626675579","responsesize":"14882","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 9 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200002,"client":"172.30.41.155:33488","multi.service_calls":0,"queuetimems":39493,"class":"HRegionServer","multi.mutations":9} 2022-03-07 04:38:00,376 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646627580376","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300000,"client":"172.30.41.118:55552","multi.service_calls":0,"queuetimems":44202,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:38:12,869 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646627592869","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:52734","multi.service_calls":0,"queuetimems":56689,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:38:32,597 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646627612596","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:55560","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 04:40:10,339 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2314ms No GCs detected 2022-03-07 04:40:40,847 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-07 04:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=650.08 KB, freeSize=306.57 MB, max=307.20 MB, blockCount=26, accesses=249907, hits=241805, hitRatio=96.76%, , cachingAccesses=241927, cachingHits=241496, cachingHitsRatio=99.82%, evictions=17633, evicted=0, evictedPerRun=0.0 2022-03-07 04:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 04:42:16,180 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646627836180","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:55596","multi.service_calls":0,"queuetimems":3,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:42:39,558 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646626659554","responsesize":"29753","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 18 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1500004,"client":"172.30.41.118:55436","multi.service_calls":0,"queuetimems":23468,"class":"HRegionServer","multi.mutations":18} 2022-03-07 04:42:45,426 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646627565425","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:33588","multi.service_calls":0,"queuetimems":29251,"class":"HRegionServer","multi.mutations":2} 2022-03-07 04:42:45,980 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646627865979","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:55596","multi.service_calls":0,"queuetimems":29802,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:42:53,919 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2243ms No GCs detected 2022-03-07 04:42:55,619 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646627875581","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300038,"client":"172.31.0.219:52752","multi.service_calls":0,"queuetimems":39404,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:43:05,535 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646627585534","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 5 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:52734","multi.service_calls":0,"queuetimems":49359,"class":"HRegionServer","multi.mutations":5} 2022-03-07 04:43:32,597 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646627912597","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:52754","multi.service_calls":0,"queuetimems":19726,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:43:36,611 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646627616610","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 8 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:55560","multi.service_calls":0,"queuetimems":4014,"class":"HRegionServer","multi.mutations":8} 2022-03-07 04:46:15,176 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2842ms No GCs detected 2022-03-07 04:46:23,560 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 6930ms No GCs detected 2022-03-07 04:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=650.08 KB, freeSize=306.57 MB, max=307.20 MB, blockCount=26, accesses=250794, hits=242692, hitRatio=96.77%, , cachingAccesses=242814, cachingHits=242383, cachingHitsRatio=99.82%, evictions=17662, evicted=0, evictedPerRun=0.0 2022-03-07 04:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 04:47:09,102 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1901ms No GCs detected 2022-03-07 04:47:16,879 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 7276ms No GCs detected 2022-03-07 04:47:18,126 INFO [regionserver/cvp328:16201.Chore.1] hbase.ScheduledChore: Chore: MemstoreFlusherChore missed its start time 2022-03-07 04:47:18,247 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646628136181","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":302066,"client":"172.30.41.118:55628","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:47:18,127 INFO [regionserver/cvp328:16201.Chore.2] hbase.ScheduledChore: Chore: CompactionChecker missed its start time 2022-03-07 04:47:25,057 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-07 04:47:39,560 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646628159558","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.118:55628","multi.service_calls":0,"queuetimems":23377,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:47:45,981 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646628165980","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:52770","multi.service_calls":0,"queuetimems":29798,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:47:55,620 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646628175620","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.31.0.219:52558","multi.service_calls":0,"queuetimems":20257,"class":"HRegionServer","multi.mutations":2} 2022-03-07 04:48:00,377 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646627880376","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 9 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:52752","multi.service_calls":0,"queuetimems":44199,"class":"HRegionServer","multi.mutations":9} 2022-03-07 04:48:05,536 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646628185536","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:55628","multi.service_calls":0,"queuetimems":26573,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:48:16,595 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646627296593","responsesize":"6622","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200002,"client":"172.30.41.155:33558","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-07 04:48:16,595 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646627296593","responsesize":"13230","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 8 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200002,"client":"172.30.41.155:33558","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":8} 2022-03-07 04:48:36,418 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 5783ms No GCs detected 2022-03-07 04:48:36,420 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646628212597","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":303823,"client":"172.31.0.219:52770","multi.service_calls":0,"queuetimems":50975,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:48:36,627 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646628216611","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300016,"client":"172.31.0.219:52770","multi.service_calls":0,"queuetimems":52779,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:50:27,363 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1564ms No GCs detected 2022-03-07 04:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=650.08 KB, freeSize=306.57 MB, max=307.20 MB, blockCount=26, accesses=251846, hits=243744, hitRatio=96.78%, , cachingAccesses=243866, cachingHits=243435, cachingHitsRatio=99.82%, evictions=17692, evicted=0, evictedPerRun=0.0 2022-03-07 04:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 04:52:18,128 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646627836176","responsesize":"19835","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 12 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":901952,"client":"172.30.41.118:55596","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":12} 2022-03-07 04:52:45,427 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646628165426","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:55628","multi.service_calls":0,"queuetimems":29244,"class":"HRegionServer","multi.mutations":3} 2022-03-07 04:53:00,378 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646628480377","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:33698","multi.service_calls":0,"queuetimems":20816,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:53:05,537 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646628485536","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:52806","multi.service_calls":0,"queuetimems":25975,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:53:12,872 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646627892871","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 8 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:55600","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":8} 2022-03-07 04:53:19,451 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646628499450","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:55676","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:53:19,986 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646627299562","responsesize":"29753","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 18 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1500424,"client":"172.30.41.118:55522","multi.service_calls":0,"queuetimems":2969,"class":"HRegionServer","multi.mutations":18} 2022-03-07 04:53:36,420 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646628516420","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:55676","multi.service_calls":0,"queuetimems":16971,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:53:36,534 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646627316532","responsesize":"24797","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 15 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1500002,"client":"172.31.0.219:52718","multi.service_calls":0,"queuetimems":19938,"class":"HRegionServer","multi.mutations":15} 2022-03-07 04:53:36,627 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646628516627","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:55676","multi.service_calls":0,"queuetimems":17178,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:56:24,188 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2219ms No GCs detected 2022-03-07 04:56:40,659 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2642ms No GCs detected 2022-03-07 04:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=676.32 KB, freeSize=306.54 MB, max=307.20 MB, blockCount=27, accesses=252813, hits=244710, hitRatio=96.79%, , cachingAccesses=244833, cachingHits=244401, cachingHitsRatio=99.82%, evictions=17722, evicted=0, evictedPerRun=0.0 2022-03-07 04:56:48,755 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 04:56:59,181 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-07 04:57:18,129 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646628738129","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:33732","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:57:18,131 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929059834 2022-03-07 04:57:18,131 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929059832 2022-03-07 04:57:18,131 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929059833 2022-03-07 04:57:18,131 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929059823 2022-03-07 04:57:18,131 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929059830 2022-03-07 04:57:18,249 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646627836177","responsesize":"9926","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 6 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1202072,"client":"172.31.0.219:52752","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":6} 2022-03-07 04:57:18,250 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929059831 2022-03-07 04:57:39,561 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646628459560","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 9 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:52806","multi.service_calls":0,"queuetimems":18404,"class":"HRegionServer","multi.mutations":9} 2022-03-07 04:57:45,982 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":1,"starttimems":"1646628465981","responsesize":"4972","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x97\\xC4\\x04tags ","processingtimems":600001,"client":"172.30.41.155:33698","multi.service_calls":0,"queuetimems":24795,"class":"HRegionServer","multi.mutations":3} 2022-03-07 04:57:45,982 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929059826 2022-03-07 04:57:45,982 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929059829 2022-03-07 04:57:45,983 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929059825 2022-03-07 04:57:45,983 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929059824 2022-03-07 04:57:55,622 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646628475621","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 4 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:52806","multi.service_calls":0,"queuetimems":16060,"class":"HRegionServer","multi.mutations":4} 2022-03-07 04:58:05,682 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646628785681","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:33736","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:58:05,849 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929059827 2022-03-07 04:58:05,849 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929059828 2022-03-07 04:58:12,873 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646628792872","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:33736","multi.service_calls":0,"queuetimems":1293,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:58:19,453 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646628799453","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:33736","multi.service_calls":0,"queuetimems":13725,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:58:19,987 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646628799986","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:33736","multi.service_calls":0,"queuetimems":8684,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:58:36,421 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646628816421","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300000,"client":"172.30.41.118:55732","multi.service_calls":0,"queuetimems":23548,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:58:36,535 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646628816534","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:55732","multi.service_calls":0,"queuetimems":22650,"class":"HRegionServer","multi.mutations":1} 2022-03-07 04:58:36,628 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646628816628","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:52842","multi.service_calls":0,"queuetimems":22301,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=748.88 KB, freeSize=306.47 MB, max=307.20 MB, blockCount=31, accesses=253869, hits=245762, hitRatio=96.81%, , cachingAccesses=245889, cachingHits=245453, cachingHitsRatio=99.82%, evictions=17752, evicted=0, evictedPerRun=0.0 2022-03-07 05:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 05:02:18,134 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646629038134","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:52856","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:02:18,135 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929060306 2022-03-07 05:02:18,137 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929060302 2022-03-07 05:02:18,137 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929060304 2022-03-07 05:02:18,137 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929060307 2022-03-07 05:02:21,537 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646628441156","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900381,"client":"172.31.0.219:52806","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 05:02:21,537 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929060314 2022-03-07 05:02:21,538 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929060303 2022-03-07 05:02:21,538 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929060305 2022-03-07 05:02:21,538 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929060309 2022-03-07 05:02:21,538 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929060301 2022-03-07 05:02:39,563 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646629059561","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.31.0.219:52856","multi.service_calls":0,"queuetimems":21427,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:02:45,983 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646629065983","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:52856","multi.service_calls":0,"queuetimems":27733,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:03:00,379 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646628780378","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 5 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":600001,"client":"172.30.41.118:55726","multi.service_calls":0,"queuetimems":42249,"class":"HRegionServer","multi.mutations":5} 2022-03-07 05:03:00,379 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929060311 2022-03-07 05:03:00,380 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929060308 2022-03-07 05:03:00,380 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929060316 2022-03-07 05:03:00,380 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929060310 2022-03-07 05:03:00,380 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929060312 2022-03-07 05:03:00,380 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929060315 2022-03-07 05:03:05,850 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646629085849","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:33760","multi.service_calls":0,"queuetimems":47599,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:03:12,917 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646629092917","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.155:33766","multi.service_calls":0,"queuetimems":2,"class":"HRegionServer","multi.mutations":2} 2022-03-07 05:03:18,145 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646628498143","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 8 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.155:33698","multi.service_calls":0,"queuetimems":38582,"class":"HRegionServer","multi.mutations":8} 2022-03-07 05:03:18,145 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929060692 2022-03-07 05:03:18,146 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929060691 2022-03-07 05:03:18,146 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929060690 2022-03-07 05:03:19,455 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646629099455","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:55780","multi.service_calls":0,"queuetimems":6581,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:03:36,422 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646629116421","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300001,"client":"172.30.41.118:55782","multi.service_calls":0,"queuetimems":23502,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:03:36,535 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646629116535","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:33768","multi.service_calls":0,"queuetimems":23658,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=748.88 KB, freeSize=306.47 MB, max=307.20 MB, blockCount=31, accesses=254881, hits=246774, hitRatio=96.82%, , cachingAccesses=246901, cachingHits=246465, cachingHitsRatio=99.82%, evictions=17782, evicted=0, evictedPerRun=0.0 2022-03-07 05:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 05:07:18,138 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646629338138","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:52872","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":2} 2022-03-07 05:07:28,356 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1695ms No GCs detected 2022-03-07 05:07:39,564 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646629359563","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:33804","multi.service_calls":0,"queuetimems":18025,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:07:45,429 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":1,"starttimems":"1646628765427","responsesize":"14622","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 9 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.155:33732","multi.service_calls":0,"queuetimems":27298,"class":"HRegionServer","multi.mutations":8} 2022-03-07 05:07:45,984 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646629365983","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:33804","multi.service_calls":0,"queuetimems":27848,"class":"HRegionServer","multi.mutations":2} 2022-03-07 05:07:55,627 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646629075626","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 6 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:55780","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":6} 2022-03-07 05:08:00,381 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646629380381","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:33812","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:08:08,696 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646629388695","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:52874","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:08:18,166 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646629398165","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300001,"client":"172.30.41.118:55830","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:08:19,457 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646629399456","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:55828","multi.service_calls":0,"queuetimems":19075,"class":"HRegionServer","multi.mutations":3} 2022-03-07 05:08:19,458 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929060313 2022-03-07 05:08:36,423 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646629416423","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:55828","multi.service_calls":0,"queuetimems":16967,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=748.88 KB, freeSize=306.47 MB, max=307.20 MB, blockCount=31, accesses=255826, hits=247719, hitRatio=96.83%, , cachingAccesses=247846, cachingHits=247410, cachingHitsRatio=99.82%, evictions=17812, evicted=0, evictedPerRun=0.0 2022-03-07 05:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 05:12:18,140 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646629638139","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:55890","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:12:21,539 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646629341538","responsesize":"11572","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 7 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:55824","multi.service_calls":0,"queuetimems":3403,"class":"HRegionServer","multi.mutations":7} 2022-03-07 05:12:39,566 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646629659565","responsesize":"11569","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:52886","multi.service_calls":0,"queuetimems":21425,"class":"HRegionServer","multi.mutations":7} 2022-03-07 05:12:45,429 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646629665429","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:55888","multi.service_calls":0,"queuetimems":27260,"class":"HRegionServer","multi.mutations":2} 2022-03-07 05:12:45,430 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929061236 2022-03-07 05:12:45,985 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646629665985","responsesize":"9917","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:55890","multi.service_calls":0,"queuetimems":27846,"class":"HRegionServer","multi.mutations":6} 2022-03-07 05:13:08,696 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646629688696","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:52886","multi.service_calls":0,"queuetimems":22711,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:13:19,459 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646629699459","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:55894","multi.service_calls":0,"queuetimems":1262,"class":"HRegionServer","multi.mutations":2} 2022-03-07 05:13:19,459 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929061624 2022-03-07 05:13:19,460 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929061625 2022-03-07 05:13:19,988 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646629099987","responsesize":"19835","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 12 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.30.41.118:55780","multi.service_calls":0,"queuetimems":7113,"class":"HRegionServer","multi.mutations":12} 2022-03-07 05:13:36,424 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646629716423","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:33854","multi.service_calls":0,"queuetimems":18256,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:13:36,630 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646629116628","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 8 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:33768","multi.service_calls":0,"queuetimems":23751,"class":"HRegionServer","multi.mutations":8} 2022-03-07 05:15:37,746 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1390ms No GCs detected 2022-03-07 05:16:44,385 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=748.88 KB, freeSize=306.47 MB, max=307.20 MB, blockCount=31, accesses=256894, hits=248787, hitRatio=96.84%, , cachingAccesses=248914, cachingHits=248478, cachingHitsRatio=99.82%, evictions=17842, evicted=0, evictedPerRun=0.0 2022-03-07 05:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 05:17:18,142 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646629938142","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300000,"client":"172.30.41.118:55960","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:17:18,143 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929061728 2022-03-07 05:17:18,143 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929061705 2022-03-07 05:17:18,143 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929061696 2022-03-07 05:17:18,143 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929061703 2022-03-07 05:17:18,252 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646629038250","responsesize":"14882","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 9 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200002,"client":"172.30.41.155:33760","multi.service_calls":0,"queuetimems":119,"class":"HRegionServer","multi.mutations":9} 2022-03-07 05:17:18,252 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929061733 2022-03-07 05:17:18,253 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929061749 2022-03-07 05:17:18,253 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929061745 2022-03-07 05:17:18,255 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929061729 2022-03-07 05:17:18,255 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929061743 2022-03-07 05:17:18,255 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929061746 2022-03-07 05:17:18,255 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929061737 2022-03-07 05:17:18,255 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929061744 2022-03-07 05:17:21,540 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646629941540","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:55960","multi.service_calls":0,"queuetimems":3399,"class":"HRegionServer","multi.mutations":3} 2022-03-07 05:17:39,567 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646629959566","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:33874","multi.service_calls":0,"queuetimems":21425,"class":"HRegionServer","multi.mutations":2} 2022-03-07 05:17:45,986 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646629965985","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:52904","multi.service_calls":0,"queuetimems":27844,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:18:08,697 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646629988696","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:52904","multi.service_calls":0,"queuetimems":47155,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:18:08,700 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929061697 2022-03-07 05:18:08,700 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929061701 2022-03-07 05:18:08,701 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929061704 2022-03-07 05:18:08,701 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929061738 2022-03-07 05:18:08,701 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929061736 2022-03-07 05:18:08,701 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929061742 2022-03-07 05:18:08,701 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929061731 2022-03-07 05:18:08,701 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929061748 2022-03-07 05:18:08,701 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929061693 2022-03-07 05:18:08,701 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929061741 2022-03-07 05:18:08,701 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929061702 2022-03-07 05:18:08,702 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929061730 2022-03-07 05:18:12,919 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646629392917","responsesize":"21487","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 13 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900002,"client":"172.31.0.219:52874","multi.service_calls":0,"queuetimems":4222,"class":"HRegionServer","multi.mutations":13} 2022-03-07 05:18:18,168 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646629698167","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:55898","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 05:18:19,989 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646629999988","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:55968","multi.service_calls":0,"queuetimems":525,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:18:19,989 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062104 2022-03-07 05:18:19,989 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929061700 2022-03-07 05:18:19,989 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929061747 2022-03-07 05:18:19,989 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929061734 2022-03-07 05:18:19,989 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929061739 2022-03-07 05:18:19,990 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929061699 2022-03-07 05:18:19,990 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929061698 2022-03-07 05:18:19,990 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929061750 2022-03-07 05:18:19,991 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929061740 2022-03-07 05:18:19,992 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929061732 2022-03-07 05:18:19,992 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929061735 2022-03-07 05:18:19,992 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062105 2022-03-07 05:18:36,541 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646629416536","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 7 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900005,"client":"172.30.41.118:55828","multi.service_calls":0,"queuetimems":36155,"class":"HRegionServer","multi.mutations":7} 2022-03-07 05:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=748.88 KB, freeSize=306.47 MB, max=307.20 MB, blockCount=31, accesses=258051, hits=249944, hitRatio=96.86%, , cachingAccesses=250071, cachingHits=249635, cachingHitsRatio=99.83%, evictions=17872, evicted=0, evictedPerRun=0.0 2022-03-07 05:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 05:22:18,145 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646630238144","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:56016","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:22:18,146 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062137 2022-03-07 05:22:18,146 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062173 2022-03-07 05:22:18,146 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062168 2022-03-07 05:22:18,146 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062175 2022-03-07 05:22:18,256 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646630238255","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:56016","multi.service_calls":0,"queuetimems":112,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:22:18,257 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062176 2022-03-07 05:22:18,257 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062178 2022-03-07 05:22:18,258 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062184 2022-03-07 05:22:18,258 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062181 2022-03-07 05:22:18,258 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062172 2022-03-07 05:22:18,259 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062171 2022-03-07 05:22:18,259 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062180 2022-03-07 05:22:18,259 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062134 2022-03-07 05:22:45,431 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646629965430","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:33874","multi.service_calls":0,"queuetimems":23890,"class":"HRegionServer","multi.mutations":2} 2022-03-07 05:22:45,431 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062157 2022-03-07 05:22:45,432 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062142 2022-03-07 05:22:45,432 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062177 2022-03-07 05:22:45,432 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062166 2022-03-07 05:22:45,432 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062138 2022-03-07 05:22:45,433 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062170 2022-03-07 05:22:45,433 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062163 2022-03-07 05:22:55,629 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646629675627","responsesize":"14879","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 9 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:33848","multi.service_calls":0,"queuetimems":37488,"class":"HRegionServer","multi.mutations":9} 2022-03-07 05:22:55,629 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062174 2022-03-07 05:22:55,630 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062136 2022-03-07 05:22:55,630 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062167 2022-03-07 05:22:55,631 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062162 2022-03-07 05:22:55,631 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062183 2022-03-07 05:23:08,785 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646630288785","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300000,"client":"172.30.41.118:56036","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:23:08,786 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062135 2022-03-07 05:23:08,786 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062160 2022-03-07 05:23:08,786 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062165 2022-03-07 05:23:08,786 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062185 2022-03-07 05:23:08,786 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062182 2022-03-07 05:23:08,787 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062161 2022-03-07 05:23:08,787 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062159 2022-03-07 05:23:08,787 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062158 2022-03-07 05:23:08,787 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062169 2022-03-07 05:23:08,787 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062179 2022-03-07 05:23:08,787 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062164 2022-03-07 05:23:19,465 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646629999464","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:33876","multi.service_calls":0,"queuetimems":2,"class":"HRegionServer","multi.mutations":2} 2022-03-07 05:23:19,996 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646630299995","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:56034","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:23:19,996 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646630299995","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:33892","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=748.88 KB, freeSize=306.47 MB, max=307.20 MB, blockCount=31, accesses=259236, hits=251129, hitRatio=96.87%, , cachingAccesses=251256, cachingHits=250820, cachingHitsRatio=99.83%, evictions=17902, evicted=0, evictedPerRun=0.0 2022-03-07 05:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 05:27:18,147 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646630538147","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:56102","multi.service_calls":0,"queuetimems":2,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:27:18,148 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062725 2022-03-07 05:27:18,149 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062726 2022-03-07 05:27:18,149 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062730 2022-03-07 05:27:18,149 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062731 2022-03-07 05:27:18,149 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062714 2022-03-07 05:27:18,149 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062724 2022-03-07 05:27:18,261 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646630538261","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:56092","multi.service_calls":0,"queuetimems":86,"class":"HRegionServer","multi.mutations":2} 2022-03-07 05:27:18,262 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062713 2022-03-07 05:27:18,262 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062722 2022-03-07 05:27:18,262 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062723 2022-03-07 05:27:39,568 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646630259567","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:52920","multi.service_calls":0,"queuetimems":21423,"class":"HRegionServer","multi.mutations":2} 2022-03-07 05:27:45,987 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646630265986","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:33888","multi.service_calls":0,"queuetimems":27843,"class":"HRegionServer","multi.mutations":4} 2022-03-07 05:28:00,384 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646629680382","responsesize":"18186","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 11 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.118:55890","multi.service_calls":0,"queuetimems":14397,"class":"HRegionServer","multi.mutations":11} 2022-03-07 05:28:00,385 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062715 2022-03-07 05:28:00,385 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062716 2022-03-07 05:28:00,385 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062727 2022-03-07 05:28:00,385 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062728 2022-03-07 05:28:08,790 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646630588789","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:33910","multi.service_calls":0,"queuetimems":13156,"class":"HRegionServer","multi.mutations":2} 2022-03-07 05:28:19,997 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646630299995","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 4 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.155:33892","multi.service_calls":0,"queuetimems":11292,"class":"HRegionServer","multi.mutations":4} 2022-03-07 05:28:19,997 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062720 2022-03-07 05:28:19,997 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062719 2022-03-07 05:28:19,997 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062729 2022-03-07 05:28:19,998 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062709 2022-03-07 05:28:36,429 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646630016428","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 7 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:33876","multi.service_calls":0,"queuetimems":16966,"class":"HRegionServer","multi.mutations":7} 2022-03-07 05:28:36,429 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929063185 2022-03-07 05:28:36,430 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062708 2022-03-07 05:28:36,430 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929063188 2022-03-07 05:28:36,431 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929063184 2022-03-07 05:28:36,431 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062706 2022-03-07 05:28:36,431 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062707 2022-03-07 05:28:36,431 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062721 2022-03-07 05:28:36,542 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646630316541","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:33892","multi.service_calls":0,"queuetimems":27838,"class":"HRegionServer","multi.mutations":6} 2022-03-07 05:28:36,631 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646630016630","responsesize":"19835","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 12 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.30.41.118:55968","multi.service_calls":0,"queuetimems":17167,"class":"HRegionServer","multi.mutations":12} 2022-03-07 05:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=748.88 KB, freeSize=306.47 MB, max=307.20 MB, blockCount=31, accesses=260214, hits=252107, hitRatio=96.88%, , cachingAccesses=252234, cachingHits=251798, cachingHitsRatio=99.83%, evictions=17932, evicted=0, evictedPerRun=0.0 2022-03-07 05:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-07 05:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 05:32:18,263 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646630838263","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:33924","multi.service_calls":0,"queuetimems":115,"class":"HRegionServer","multi.mutations":3} 2022-03-07 05:32:18,264 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929063186 2022-03-07 05:32:18,265 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929063189 2022-03-07 05:32:21,542 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646630241540","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.155:33888","multi.service_calls":0,"queuetimems":3397,"class":"HRegionServer","multi.mutations":4} 2022-03-07 05:32:55,636 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646630575635","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:56106","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-07 05:33:18,236 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646630898236","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:56186","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 05:33:18,237 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929063187 2022-03-07 05:33:18,237 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929063250 2022-03-07 05:33:19,466 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646630599465","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 6 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:56106","multi.service_calls":0,"queuetimems":23830,"class":"HRegionServer","multi.mutations":6} 2022-03-07 05:33:19,997 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646630599996","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:52944","multi.service_calls":0,"queuetimems":24362,"class":"HRegionServer","multi.mutations":2} 2022-03-07 05:33:36,432 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646630916431","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:33930","multi.service_calls":0,"queuetimems":16433,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:33:36,543 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646630916542","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:52962","multi.service_calls":0,"queuetimems":36157,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=748.88 KB, freeSize=306.47 MB, max=307.20 MB, blockCount=31, accesses=261226, hits=253119, hitRatio=96.90%, , cachingAccesses=253246, cachingHits=252810, cachingHitsRatio=99.83%, evictions=17962, evicted=0, evictedPerRun=0.0 2022-03-07 05:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 05:37:18,151 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646630838149","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:56170","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":2} 2022-03-07 05:37:18,152 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062717 2022-03-07 05:37:18,152 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062704 2022-03-07 05:37:18,152 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929062718 2022-03-07 05:37:18,152 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929063281 2022-03-07 05:37:55,637 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646631175637","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:56256","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:37:55,638 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929063280 2022-03-07 05:38:18,238 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646631198237","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:56256","multi.service_calls":0,"queuetimems":22600,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:38:19,466 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646631199466","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:56256","multi.service_calls":0,"queuetimems":23829,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:38:19,999 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646631199997","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.118:56256","multi.service_calls":0,"queuetimems":24360,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:38:19,999 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646630899998","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:33930","multi.service_calls":0,"queuetimems":19613,"class":"HRegionServer","multi.mutations":3} 2022-03-07 05:38:36,432 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646631216432","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:56256","multi.service_calls":0,"queuetimems":40795,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:38:36,543 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646631216543","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:56256","multi.service_calls":0,"queuetimems":40906,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:38:36,632 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646630916631","responsesize":"16528","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 10 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":600001,"client":"172.31.0.219:52962","multi.service_calls":0,"queuetimems":36246,"class":"HRegionServer","multi.mutations":10} 2022-03-07 05:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=748.88 KB, freeSize=306.47 MB, max=307.20 MB, blockCount=31, accesses=262100, hits=253993, hitRatio=96.91%, , cachingAccesses=254120, cachingHits=253684, cachingHitsRatio=99.83%, evictions=17992, evicted=0, evictedPerRun=0.0 2022-03-07 05:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 05:42:18,153 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646631438152","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:52996","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:42:18,266 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646631138265","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:56252","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":3} 2022-03-07 05:42:21,543 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646631141542","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:52980","multi.service_calls":0,"queuetimems":3277,"class":"HRegionServer","multi.mutations":2} 2022-03-07 05:42:39,572 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646630859568","responsesize":"18183","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 11 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900004,"client":"172.30.41.118:56170","multi.service_calls":0,"queuetimems":21419,"class":"HRegionServer","multi.mutations":11} 2022-03-07 05:42:45,435 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646630565433","responsesize":"18186","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 11 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.118:56102","multi.service_calls":0,"queuetimems":27287,"class":"HRegionServer","multi.mutations":11} 2022-03-07 05:42:45,989 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646630865987","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 7 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.31.0.219:52958","multi.service_calls":0,"queuetimems":27838,"class":"HRegionServer","multi.mutations":7} 2022-03-07 05:42:55,638 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646631475638","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:56322","multi.service_calls":0,"queuetimems":37486,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:43:00,387 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646630880385","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.155:33930","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 05:43:19,999 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646630599997","responsesize":"19838","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 12 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.31.0.219:52944","multi.service_calls":0,"queuetimems":24363,"class":"HRegionServer","multi.mutations":12} 2022-03-07 05:43:36,433 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646631516433","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300000,"client":"172.30.41.118:56322","multi.service_calls":0,"queuetimems":27313,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:43:36,544 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646631516543","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:56322","multi.service_calls":0,"queuetimems":40905,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:44:41,600 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1329ms No GCs detected 2022-03-07 05:44:57,006 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-07 05:46:36,120 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4830ms No GCs detected 2022-03-07 05:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=748.88 KB, freeSize=306.47 MB, max=307.20 MB, blockCount=31, accesses=263015, hits=254908, hitRatio=96.92%, , cachingAccesses=255035, cachingHits=254599, cachingHitsRatio=99.83%, evictions=18021, evicted=0, evictedPerRun=0.0 2022-03-07 05:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 05:47:18,154 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646631738154","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:56394","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:47:21,543 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646631741543","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:33986","multi.service_calls":0,"queuetimems":3389,"class":"HRegionServer","multi.mutations":3} 2022-03-07 05:47:45,436 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646631765435","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:53016","multi.service_calls":0,"queuetimems":27281,"class":"HRegionServer","multi.mutations":2} 2022-03-07 05:47:45,989 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646631765989","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:53016","multi.service_calls":0,"queuetimems":27834,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:47:55,640 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646631775640","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:56416","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:48:00,389 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646631780388","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:56416","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:48:18,239 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646631498238","responsesize":"16528","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 10 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":600001,"client":"172.31.0.219:52996","multi.service_calls":0,"queuetimems":60086,"class":"HRegionServer","multi.mutations":10} 2022-03-07 05:48:19,468 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646631499467","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:56322","multi.service_calls":0,"queuetimems":61315,"class":"HRegionServer","multi.mutations":4} 2022-03-07 05:48:20,000 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646631499999","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 6 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:33964","multi.service_calls":0,"queuetimems":24361,"class":"HRegionServer","multi.mutations":6} 2022-03-07 05:48:36,544 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646631816544","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:56416","multi.service_calls":0,"queuetimems":40904,"class":"HRegionServer","multi.mutations":2} 2022-03-07 05:50:11,846 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1567ms No GCs detected 2022-03-07 05:50:28,854 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-07 05:51:06,031 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1579ms No GCs detected 2022-03-07 05:51:19,197 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1531ms No GCs detected 2022-03-07 05:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=748.88 KB, freeSize=306.47 MB, max=307.20 MB, blockCount=31, accesses=263796, hits=255689, hitRatio=96.93%, , cachingAccesses=255816, cachingHits=255380, cachingHitsRatio=99.83%, evictions=18051, evicted=0, evictedPerRun=0.0 2022-03-07 05:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 05:52:18,156 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646632038156","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:34002","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:52:18,267 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646631738266","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:56394","multi.service_calls":0,"queuetimems":112,"class":"HRegionServer","multi.mutations":4} 2022-03-07 05:52:21,544 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646632041544","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:34002","multi.service_calls":0,"queuetimems":3388,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:52:45,436 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646632065436","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:34002","multi.service_calls":0,"queuetimems":27280,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:52:45,990 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646632065990","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:34002","multi.service_calls":0,"queuetimems":27834,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:53:00,390 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646632080389","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:56476","multi.service_calls":0,"queuetimems":38845,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:53:18,241 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646632098240","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:56492","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:53:20,001 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646631799999","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:33988","multi.service_calls":0,"queuetimems":19611,"class":"HRegionServer","multi.mutations":3} 2022-03-07 05:53:20,002 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646631499999","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 5 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.155:33964","multi.service_calls":0,"queuetimems":61847,"class":"HRegionServer","multi.mutations":5} 2022-03-07 05:53:36,434 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646631816433","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:33988","multi.service_calls":0,"queuetimems":36045,"class":"HRegionServer","multi.mutations":3} 2022-03-07 05:53:36,545 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646632116545","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:56488","multi.service_calls":0,"queuetimems":18275,"class":"HRegionServer","multi.mutations":2} 2022-03-07 05:53:36,634 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646631516632","responsesize":"19835","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 12 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900002,"client":"172.30.41.118:56322","multi.service_calls":0,"queuetimems":40994,"class":"HRegionServer","multi.mutations":12} 2022-03-07 05:54:04,227 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1514ms No GCs detected 2022-03-07 05:55:54,534 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1534ms No GCs detected 2022-03-07 05:55:57,944 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2410ms No GCs detected 2022-03-07 05:56:04,394 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-07 05:56:04,635 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 576ms. 2022-03-07 05:56:14,022 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1089ms No GCs detected 2022-03-07 05:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=748.88 KB, freeSize=306.47 MB, max=307.20 MB, blockCount=31, accesses=264585, hits=256478, hitRatio=96.94%, , cachingAccesses=256605, cachingHits=256169, cachingHitsRatio=99.83%, evictions=18081, evicted=0, evictedPerRun=0.0 2022-03-07 05:56:46,992 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: java.net.ConnectException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-07 05:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 05:57:23,452 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1206ms No GCs detected 2022-03-07 05:57:39,575 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646631759572","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900003,"client":"172.30.41.155:33986","multi.service_calls":0,"queuetimems":21418,"class":"HRegionServer","multi.mutations":5} 2022-03-07 05:57:45,437 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646632365436","responsesize":"9917","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:56542","multi.service_calls":0,"queuetimems":27279,"class":"HRegionServer","multi.mutations":6} 2022-03-07 05:57:59,517 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1379ms No GCs detected 2022-03-07 05:58:02,239 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1221ms No GCs detected 2022-03-07 05:58:18,241 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646632398241","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:34024","multi.service_calls":0,"queuetimems":17850,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:58:20,002 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646632400001","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:34024","multi.service_calls":0,"queuetimems":19610,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:58:36,435 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646632416434","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:34024","multi.service_calls":0,"queuetimems":36043,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:58:36,546 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646632416545","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:53058","multi.service_calls":0,"queuetimems":36154,"class":"HRegionServer","multi.mutations":1} 2022-03-07 05:58:36,634 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646632416634","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:53058","multi.service_calls":0,"queuetimems":36243,"class":"HRegionServer","multi.mutations":2} 2022-03-07 06:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=748.88 KB, freeSize=306.47 MB, max=307.20 MB, blockCount=31, accesses=265255, hits=257148, hitRatio=96.94%, , cachingAccesses=257275, cachingHits=256839, cachingHitsRatio=99.83%, evictions=18111, evicted=0, evictedPerRun=0.0 2022-03-07 06:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 06:02:18,158 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646632338157","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:56542","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-07 06:02:18,269 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646632338268","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:34020","multi.service_calls":0,"queuetimems":110,"class":"HRegionServer","multi.mutations":2} 2022-03-07 06:02:21,546 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646632341545","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":600000,"client":"172.30.41.118:56542","multi.service_calls":0,"queuetimems":3277,"class":"HRegionServer","multi.mutations":2} 2022-03-07 06:03:00,391 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646632380390","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:56560","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-07 06:03:19,469 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646632099468","responsesize":"21487","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 13 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.31.0.219:53040","multi.service_calls":0,"queuetimems":1226,"class":"HRegionServer","multi.mutations":13} 2022-03-07 06:03:20,002 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646632100000","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.155:34008","multi.service_calls":0,"queuetimems":1760,"class":"HRegionServer","multi.mutations":3} 2022-03-07 06:03:20,196 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646632700195","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:56650","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:03:33,041 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1232ms No GCs detected 2022-03-07 06:03:36,435 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646632716435","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:56650","multi.service_calls":0,"queuetimems":16240,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:03:36,546 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646632716546","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300000,"client":"172.30.41.118:56650","multi.service_calls":0,"queuetimems":16351,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:03:36,635 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646632716634","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300001,"client":"172.30.41.118:56648","multi.service_calls":0,"queuetimems":16329,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:06:33,738 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1001ms No GCs detected 2022-03-07 06:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=748.88 KB, freeSize=306.47 MB, max=307.20 MB, blockCount=31, accesses=266035, hits=257928, hitRatio=96.95%, , cachingAccesses=258055, cachingHits=257619, cachingHitsRatio=99.83%, evictions=18141, evicted=0, evictedPerRun=0.0 2022-03-07 06:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 06:07:04,775 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2031ms No GCs detected 2022-03-07 06:07:08,654 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2378ms No GCs detected 2022-03-07 06:07:18,163 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646632938162","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:56694","multi.service_calls":0,"queuetimems":3,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:07:18,270 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646632938269","responsesize":"9917","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:56694","multi.service_calls":0,"queuetimems":110,"class":"HRegionServer","multi.mutations":6} 2022-03-07 06:07:39,588 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646632659576","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600012,"client":"172.30.41.155:34038","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-07 06:07:45,995 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646632365990","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 7 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900005,"client":"172.30.41.118:56542","multi.service_calls":0,"queuetimems":27722,"class":"HRegionServer","multi.mutations":7} 2022-03-07 06:08:00,393 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646632980392","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:34054","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:08:18,242 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646632698241","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:53074","multi.service_calls":0,"queuetimems":38654,"class":"HRegionServer","multi.mutations":3} 2022-03-07 06:08:19,470 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646632999470","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:56712","multi.service_calls":0,"queuetimems":19078,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:08:20,003 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646632400002","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 5 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:56560","multi.service_calls":0,"queuetimems":19611,"class":"HRegionServer","multi.mutations":5} 2022-03-07 06:08:20,004 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646633000002","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.155:34054","multi.service_calls":0,"queuetimems":19610,"class":"HRegionServer","multi.mutations":3} 2022-03-07 06:08:20,220 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646633000196","responsesize":"11569","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300024,"client":"172.31.0.219:53092","multi.service_calls":0,"queuetimems":19803,"class":"HRegionServer","multi.mutations":7} 2022-03-07 06:08:28,536 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-07 06:10:11,502 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1016ms No GCs detected 2022-03-07 06:10:34,296 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-07 06:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=748.88 KB, freeSize=306.47 MB, max=307.20 MB, blockCount=31, accesses=266748, hits=258641, hitRatio=96.96%, , cachingAccesses=258768, cachingHits=258332, cachingHitsRatio=99.83%, evictions=18171, evicted=0, evictedPerRun=0.0 2022-03-07 06:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 06:12:18,164 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646633238164","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300000,"client":"172.30.41.118:56766","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 06:12:18,271 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646633238270","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:56766","multi.service_calls":0,"queuetimems":106,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:12:39,588 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646633259588","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:34068","multi.service_calls":0,"queuetimems":21424,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:12:45,438 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646632665437","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 6 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":900001,"client":"172.30.41.118:56632","multi.service_calls":0,"queuetimems":5861,"class":"HRegionServer","multi.mutations":6} 2022-03-07 06:12:45,995 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646633265995","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:34068","multi.service_calls":0,"queuetimems":27831,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:12:55,642 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646632075640","responsesize":"29753","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 18 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1500002,"client":"172.30.41.118:56476","multi.service_calls":0,"queuetimems":34095,"class":"HRegionServer","multi.mutations":18} 2022-03-07 06:13:00,394 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646633280394","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:34074","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 06:13:18,243 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646633298242","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:34074","multi.service_calls":0,"queuetimems":17848,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:13:20,004 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646633300003","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:53112","multi.service_calls":0,"queuetimems":531,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:13:20,005 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646633300004","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:53112","multi.service_calls":0,"queuetimems":19610,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:13:41,135 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646633016547","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":604588,"client":"172.30.41.118:56712","multi.service_calls":0,"queuetimems":36155,"class":"HRegionServer","multi.mutations":4} 2022-03-07 06:13:44,102 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 6744ms No GCs detected 2022-03-07 06:13:46,994 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2391ms No GCs detected 2022-03-07 06:13:50,740 WARN [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Scan(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ScanRequest)","starttimems":"1646633616857","responsesize":"738","method":"Scan","param":"region { type: REGION_NAME value: \"hbase:meta,,1\" } scan { column { family: \"info\" } start_row: \"aeris_v2,,:\" stop_row: \"aeris_v2\" time_range { } max_ ","processingtimems":10214,"client":"172.31.0.219:53126","queuetimems":1,"class":"HRegionServer","scandetails":"table: hbase:meta region: hbase:meta,,1.1588230740"} 2022-03-07 06:14:05,482 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1043ms No GCs detected 2022-03-07 06:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=748.88 KB, freeSize=306.47 MB, max=307.20 MB, blockCount=31, accesses=267455, hits=259348, hitRatio=96.97%, , cachingAccesses=259475, cachingHits=259039, cachingHitsRatio=99.83%, evictions=18200, evicted=0, evictedPerRun=0.0 2022-03-07 06:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 06:17:18,166 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646633538165","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:56832","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:17:18,272 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646633538271","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:56832","multi.service_calls":0,"queuetimems":105,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:17:21,811 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646632941809","responsesize":"14879","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 9 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.155:34050","multi.service_calls":0,"queuetimems":3649,"class":"HRegionServer","multi.mutations":9} 2022-03-07 06:17:45,439 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646633565438","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:56832","multi.service_calls":0,"queuetimems":27272,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:17:45,996 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646633565995","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:56832","multi.service_calls":0,"queuetimems":27829,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:18:00,395 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646633580394","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:53122","multi.service_calls":0,"queuetimems":42228,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:18:18,244 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646633598243","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:56848","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:18:20,149 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646633600005","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300144,"client":"172.30.41.155:34090","multi.service_calls":0,"queuetimems":1761,"class":"HRegionServer","multi.mutations":2} 2022-03-07 06:18:36,437 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646633016435","responsesize":"21487","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 13 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900002,"client":"172.30.41.118:56712","multi.service_calls":0,"queuetimems":16965,"class":"HRegionServer","multi.mutations":13} 2022-03-07 06:18:41,136 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646633621135","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:34090","multi.service_calls":0,"queuetimems":22891,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:19:11,389 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1195ms No GCs detected 2022-03-07 06:19:55,405 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2703ms No GCs detected 2022-03-07 06:20:01,246 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 5341ms No GCs detected 2022-03-07 06:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=807.45 KB, freeSize=306.41 MB, max=307.20 MB, blockCount=32, accesses=268074, hits=259966, hitRatio=96.98%, , cachingAccesses=260094, cachingHits=259657, cachingHitsRatio=99.83%, evictions=18230, evicted=0, evictedPerRun=0.0 2022-03-07 06:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 06:22:18,168 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646633838168","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:34104","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:22:18,273 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646633838272","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300000,"client":"172.30.41.118:56900","multi.service_calls":0,"queuetimems":105,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:22:21,811 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646633841811","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:34104","multi.service_calls":0,"queuetimems":3644,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:22:39,590 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646633559588","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.155:34086","multi.service_calls":0,"queuetimems":21422,"class":"HRegionServer","multi.mutations":4} 2022-03-07 06:22:45,439 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646633865439","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:34104","multi.service_calls":0,"queuetimems":27272,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:23:18,245 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646633898245","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:56920","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:23:20,006 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646633600005","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:56848","multi.service_calls":0,"queuetimems":1761,"class":"HRegionServer","multi.mutations":4} 2022-03-07 06:23:20,165 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646633900149","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300016,"client":"172.31.0.219:53146","multi.service_calls":0,"queuetimems":1904,"class":"HRegionServer","multi.mutations":3} 2022-03-07 06:23:20,222 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646633300220","responsesize":"19835","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 12 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900002,"client":"172.31.0.219:53112","multi.service_calls":0,"queuetimems":19825,"class":"HRegionServer","multi.mutations":12} 2022-03-07 06:23:36,437 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646633916437","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:53146","multi.service_calls":0,"queuetimems":18192,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:23:40,068 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646633016635","responsesize":"13230","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 8 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1203433,"client":"172.31.0.219:53092","multi.service_calls":0,"queuetimems":36242,"class":"HRegionServer","multi.mutations":8} 2022-03-07 06:23:40,068 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929067632 2022-03-07 06:23:41,137 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646633921136","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:34108","multi.service_calls":0,"queuetimems":22891,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=268815, hits=260705, hitRatio=96.98%, , cachingAccesses=260835, cachingHits=260396, cachingHitsRatio=99.83%, evictions=18260, evicted=0, evictedPerRun=0.0 2022-03-07 06:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 06:27:18,170 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646634138169","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:56968","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:27:55,644 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646633575643","responsesize":"19835","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 12 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.31.0.219:53122","multi.service_calls":0,"queuetimems":37359,"class":"HRegionServer","multi.mutations":12} 2022-03-07 06:28:02,274 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646633882272","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:53140","multi.service_calls":0,"queuetimems":44105,"class":"HRegionServer","multi.mutations":3} 2022-03-07 06:28:18,247 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646634198246","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:56986","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:28:20,222 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646634200222","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:53160","multi.service_calls":0,"queuetimems":1975,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:28:36,438 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646634216438","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:53160","multi.service_calls":0,"queuetimems":18191,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:28:40,069 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646634220068","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:53160","multi.service_calls":0,"queuetimems":21822,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:28:41,138 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646634221137","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:53160","multi.service_calls":0,"queuetimems":22891,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=269634, hits=261524, hitRatio=96.99%, , cachingAccesses=261654, cachingHits=261215, cachingHitsRatio=99.83%, evictions=18290, evicted=0, evictedPerRun=0.0 2022-03-07 06:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-07 06:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 06:32:18,171 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646634438171","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:57046","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 06:32:18,172 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929068023 2022-03-07 06:32:18,276 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":2,"starttimems":"1646634138273","responsesize":"9406","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 6 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.155:34116","multi.service_calls":0,"queuetimems":104,"class":"HRegionServer","multi.mutations":4} 2022-03-07 06:32:21,813 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":1,"starttimems":"1646634141812","responsesize":"9662","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 6 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:34116","multi.service_calls":0,"queuetimems":3643,"class":"HRegionServer","multi.mutations":5} 2022-03-07 06:32:45,553 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646634165439","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600113,"client":"172.31.0.219:53156","multi.service_calls":0,"queuetimems":27270,"class":"HRegionServer","multi.mutations":3} 2022-03-07 06:32:45,567 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3253ms No GCs detected 2022-03-07 06:33:02,274 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646634482274","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:57046","multi.service_calls":0,"queuetimems":44103,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:33:18,277 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646634498247","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300030,"client":"172.30.41.118:57046","multi.service_calls":0,"queuetimems":37990,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:33:19,474 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646633299471","responsesize":"31405","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 19 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1500002,"client":"172.30.41.118:56778","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":19} 2022-03-07 06:33:20,211 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":1,"starttimems":"1646634200124","responsesize":"4972","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 4 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600087,"client":"172.30.41.155:34116","multi.service_calls":0,"queuetimems":61851,"class":"HRegionServer","multi.mutations":3} 2022-03-07 06:33:20,223 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646634500223","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:34132","multi.service_calls":0,"queuetimems":37461,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:33:26,973 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1090ms No GCs detected 2022-03-07 06:33:30,636 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1008ms No GCs detected 2022-03-07 06:33:34,575 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1277ms No GCs detected 2022-03-07 06:33:37,603 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646634516438","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":301165,"client":"172.30.41.155:34132","multi.service_calls":0,"queuetimems":49482,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:33:40,070 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646634520069","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:34132","multi.service_calls":0,"queuetimems":63050,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:33:41,138 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646634521138","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:34132","multi.service_calls":0,"queuetimems":58086,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:34:09,015 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-07 06:34:48,000 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-07 06:35:45,490 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3611ms No GCs detected 2022-03-07 06:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=270439, hits=262329, hitRatio=97.00%, , cachingAccesses=262459, cachingHits=262020, cachingHitsRatio=99.83%, evictions=18319, evicted=0, evictedPerRun=0.0 2022-03-07 06:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 06:37:18,173 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646634738172","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:57116","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 06:37:18,174 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929068464 2022-03-07 06:37:45,553 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646634765553","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:57116","multi.service_calls":0,"queuetimems":27381,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:37:46,709 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646633865996","responsesize":"14882","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 9 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200713,"client":"172.30.41.118:56900","multi.service_calls":0,"queuetimems":27829,"class":"HRegionServer","multi.mutations":9} 2022-03-07 06:37:55,645 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646634475645","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.31.0.219:53174","multi.service_calls":0,"queuetimems":37474,"class":"HRegionServer","multi.mutations":2} 2022-03-07 06:38:18,278 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646634798277","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:53214","multi.service_calls":0,"queuetimems":16002,"class":"HRegionServer","multi.mutations":2} 2022-03-07 06:38:19,475 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646634799474","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:34150","multi.service_calls":0,"queuetimems":17199,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:38:20,166 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646634200165","responsesize":"14879","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 9 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:56986","multi.service_calls":0,"queuetimems":1919,"class":"HRegionServer","multi.mutations":9} 2022-03-07 06:38:20,213 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646634800211","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.118:57124","multi.service_calls":0,"queuetimems":737,"class":"HRegionServer","multi.mutations":3} 2022-03-07 06:38:40,070 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646634820070","responsesize":"6613","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:57124","multi.service_calls":0,"queuetimems":20595,"class":"HRegionServer","multi.mutations":4} 2022-03-07 06:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=271278, hits=263168, hitRatio=97.01%, , cachingAccesses=263298, cachingHits=262859, cachingHitsRatio=99.83%, evictions=18349, evicted=0, evictedPerRun=0.0 2022-03-07 06:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 06:42:21,814 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646634741813","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:53208","multi.service_calls":0,"queuetimems":3640,"class":"HRegionServer","multi.mutations":2} 2022-03-07 06:42:39,595 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646634159591","responsesize":"26446","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 16 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200003,"client":"172.30.41.118:56968","multi.service_calls":0,"queuetimems":21422,"class":"HRegionServer","multi.mutations":16} 2022-03-07 06:43:02,276 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646634782275","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 4 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:34150","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-07 06:43:18,278 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646635098278","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:34164","multi.service_calls":0,"queuetimems":60104,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:43:19,475 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646635099475","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:34164","multi.service_calls":0,"queuetimems":32765,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:43:20,167 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646635100166","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:57172","multi.service_calls":0,"queuetimems":33456,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:43:20,214 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646635100213","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:34164","multi.service_calls":0,"queuetimems":33503,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:43:20,224 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646634800223","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:53214","multi.service_calls":0,"queuetimems":17948,"class":"HRegionServer","multi.mutations":2} 2022-03-07 06:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=272159, hits=264049, hitRatio=97.02%, , cachingAccesses=264179, cachingHits=263740, cachingHitsRatio=99.83%, evictions=18379, evicted=0, evictedPerRun=0.0 2022-03-07 06:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 06:47:18,175 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646635038174","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:57172","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 06:47:18,278 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":2,"starttimems":"1646634738276","responsesize":"11591","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 9 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x97\\xC4\\x04tags ","processingtimems":900002,"client":"172.30.41.155:34146","multi.service_calls":0,"queuetimems":104,"class":"HRegionServer","multi.mutations":7} 2022-03-07 06:47:45,554 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646635065554","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:57172","multi.service_calls":0,"queuetimems":27380,"class":"HRegionServer","multi.mutations":3} 2022-03-07 06:47:55,647 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646635075646","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 4 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:53240","multi.service_calls":0,"queuetimems":37472,"class":"HRegionServer","multi.mutations":4} 2022-03-07 06:48:19,476 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":2,"starttimems":"1646635399475","responsesize":"1675","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x97\\xC4\\x04tags ","processingtimems":300001,"client":"172.30.41.155:34194","multi.service_calls":0,"queuetimems":57660,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:48:20,168 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646635400167","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:57260","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:48:20,214 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646635400214","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:34194","multi.service_calls":0,"queuetimems":40618,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:48:37,604 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646634817603","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:57124","multi.service_calls":0,"queuetimems":35328,"class":"HRegionServer","multi.mutations":4} 2022-03-07 06:48:40,071 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646635120071","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.31.0.219:53240","multi.service_calls":0,"queuetimems":53361,"class":"HRegionServer","multi.mutations":2} 2022-03-07 06:48:41,140 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646634821138","responsesize":"19835","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 12 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900002,"client":"172.31.0.219:53214","multi.service_calls":0,"queuetimems":21663,"class":"HRegionServer","multi.mutations":12} 2022-03-07 06:51:24,286 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1662ms No GCs detected 2022-03-07 06:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=272847, hits=264737, hitRatio=97.03%, , cachingAccesses=264867, cachingHits=264428, cachingHitsRatio=99.83%, evictions=18409, evicted=0, evictedPerRun=0.0 2022-03-07 06:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 06:52:18,177 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646635638176","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:57304","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:52:18,279 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646635638278","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:34234","multi.service_calls":0,"queuetimems":102,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:52:18,279 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929069705 2022-03-07 06:52:45,555 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646635665555","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:34234","multi.service_calls":0,"queuetimems":27378,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:52:55,647 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646635675647","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:53300","multi.service_calls":0,"queuetimems":37470,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:53:02,277 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646635382276","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:34194","multi.service_calls":0,"queuetimems":40461,"class":"HRegionServer","multi.mutations":3} 2022-03-07 06:53:20,216 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646635700215","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:53306","multi.service_calls":0,"queuetimems":46,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:53:20,225 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646635400224","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 9 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:53276","multi.service_calls":0,"queuetimems":56,"class":"HRegionServer","multi.mutations":9} 2022-03-07 06:53:40,072 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646635720071","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:53306","multi.service_calls":0,"queuetimems":19902,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:53:41,141 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646635721140","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:34244","multi.service_calls":0,"queuetimems":3857,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=273610, hits=265500, hitRatio=97.04%, , cachingAccesses=265630, cachingHits=265191, cachingHitsRatio=99.83%, evictions=18439, evicted=0, evictedPerRun=0.0 2022-03-07 06:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 06:57:18,178 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646635938178","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:57362","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:57:46,714 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":2,"starttimems":"1646635066710","responsesize":"16019","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 10 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200004,"client":"172.30.41.155:34164","multi.service_calls":0,"queuetimems":28536,"class":"HRegionServer","multi.mutations":8} 2022-03-07 06:57:55,648 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646635975647","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:53314","multi.service_calls":0,"queuetimems":37469,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:58:02,278 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646635982277","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:53314","multi.service_calls":0,"queuetimems":44099,"class":"HRegionServer","multi.mutations":2} 2022-03-07 06:58:18,280 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646635398278","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 7 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.155:34194","multi.service_calls":0,"queuetimems":38682,"class":"HRegionServer","multi.mutations":7} 2022-03-07 06:58:19,478 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646635699477","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:57316","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 06:58:20,169 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646635700168","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:57316","multi.service_calls":0,"queuetimems":691,"class":"HRegionServer","multi.mutations":4} 2022-03-07 06:58:20,226 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":1,"starttimems":"1646636000225","responsesize":"1665","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x97\\xC4\\x04tags ","processingtimems":300001,"client":"172.30.41.155:34262","multi.service_calls":0,"queuetimems":8,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:58:20,217 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646636000217","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:57372","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:58:28,654 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-07 06:58:43,975 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1999ms No GCs detected 2022-03-07 06:58:43,975 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646636021141","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":302834,"client":"172.30.41.155:34262","multi.service_calls":0,"queuetimems":20915,"class":"HRegionServer","multi.mutations":1} 2022-03-07 06:59:10,866 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1026ms No GCs detected 2022-03-07 07:00:03,226 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1018ms No GCs detected 2022-03-07 07:00:11,924 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1697ms No GCs detected 2022-03-07 07:00:32,366 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-07 07:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=274210, hits=266100, hitRatio=97.04%, , cachingAccesses=266230, cachingHits=265791, cachingHitsRatio=99.84%, evictions=18469, evicted=0, evictedPerRun=0.0 2022-03-07 07:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 07:02:18,180 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646636238179","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:57434","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:02:21,816 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646635341815","responsesize":"6622","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200001,"client":"172.30.41.118:57242","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-07 07:02:46,715 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":1,"starttimems":"1646636266714","responsesize":"1667","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x97\\xC4\\x04tags ","processingtimems":300001,"client":"172.30.41.155:34278","multi.service_calls":0,"queuetimems":28535,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:03:14,652 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1316ms No GCs detected 2022-03-07 07:03:18,280 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646636298280","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:57434","multi.service_calls":0,"queuetimems":31561,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:03:19,479 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646636299478","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:57434","multi.service_calls":0,"queuetimems":32759,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:03:20,395 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646636300394","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:57434","multi.service_calls":0,"queuetimems":33675,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:03:20,395 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646636300395","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:57434","multi.service_calls":0,"queuetimems":33676,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:03:20,395 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646636300395","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:57434","multi.service_calls":0,"queuetimems":33676,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:03:37,606 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646635717604","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 5 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:53306","multi.service_calls":0,"queuetimems":18126,"class":"HRegionServer","multi.mutations":5} 2022-03-07 07:03:49,218 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646636329218","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:57456","multi.service_calls":0,"queuetimems":6,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=275019, hits=266909, hitRatio=97.05%, , cachingAccesses=267039, cachingHits=266600, cachingHitsRatio=99.84%, evictions=18499, evicted=0, evictedPerRun=0.0 2022-03-07 07:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 07:07:18,281 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646635938279","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 4 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:34258","multi.service_calls":0,"queuetimems":101,"class":"HRegionServer","multi.mutations":4} 2022-03-07 07:07:21,817 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":1,"starttimems":"1646636541817","responsesize":"1667","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x97\\xC4\\x04tags ","processingtimems":300000,"client":"172.30.41.155:34290","multi.service_calls":0,"queuetimems":3636,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:07:39,600 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646635359595","responsesize":"24797","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 15 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1500004,"client":"172.30.41.118:57242","multi.service_calls":0,"queuetimems":17780,"class":"HRegionServer","multi.mutations":15} 2022-03-07 07:08:20,417 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646636600415","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.155:34294","multi.service_calls":0,"queuetimems":19,"class":"HRegionServer","multi.mutations":3} 2022-03-07 07:08:37,606 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646636617606","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:57506","multi.service_calls":0,"queuetimems":17191,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:08:49,219 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646636629218","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:53364","multi.service_calls":0,"queuetimems":28822,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:10:09,104 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1123ms No GCs detected 2022-03-07 07:10:14,108 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3004ms No GCs detected 2022-03-07 07:10:46,412 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-07 07:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=275684, hits=267574, hitRatio=97.06%, , cachingAccesses=267704, cachingHits=267265, cachingHitsRatio=99.84%, evictions=18529, evicted=0, evictedPerRun=0.0 2022-03-07 07:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 07:12:18,282 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646636838282","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:57566","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:12:45,557 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646635965555","responsesize":"24794","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 15 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.118:57362","multi.service_calls":0,"queuetimems":27377,"class":"HRegionServer","multi.mutations":15} 2022-03-07 07:12:46,716 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646636566715","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:53348","multi.service_calls":0,"queuetimems":28534,"class":"HRegionServer","multi.mutations":4} 2022-03-07 07:12:55,653 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646636275648","responsesize":"19835","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 12 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900005,"client":"172.30.41.118:57434","multi.service_calls":0,"queuetimems":37469,"class":"HRegionServer","multi.mutations":12} 2022-03-07 07:13:18,281 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646636598280","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:53348","multi.service_calls":0,"queuetimems":56463,"class":"HRegionServer","multi.mutations":4} 2022-03-07 07:13:19,480 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646636599479","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 8 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:53348","multi.service_calls":0,"queuetimems":61298,"class":"HRegionServer","multi.mutations":8} 2022-03-07 07:13:20,397 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646636600396","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 6 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:57506","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":6} 2022-03-07 07:13:20,674 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646636600415","responsesize":"16528","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 10 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":600258,"client":"172.31.0.219:53364","multi.service_calls":0,"queuetimems":19,"class":"HRegionServer","multi.mutations":10} 2022-03-07 07:13:20,675 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646636900674","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:34308","multi.service_calls":0,"queuetimems":41073,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:13:40,976 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646636020072","responsesize":"24794","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 15 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200904,"client":"172.30.41.118:57372","multi.service_calls":0,"queuetimems":19855,"class":"HRegionServer","multi.mutations":15} 2022-03-07 07:13:49,219 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646636929219","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:53398","multi.service_calls":0,"queuetimems":11612,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:13:53,202 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1002ms No GCs detected 2022-03-07 07:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=276435, hits=268325, hitRatio=97.07%, , cachingAccesses=268455, cachingHits=268016, cachingHitsRatio=99.84%, evictions=18559, evicted=0, evictedPerRun=0.0 2022-03-07 07:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 07:17:18,182 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646636538181","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 4 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:34290","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-07 07:17:21,820 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646636841818","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:53388","multi.service_calls":0,"queuetimems":3536,"class":"HRegionServer","multi.mutations":4} 2022-03-07 07:17:45,558 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646637165558","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:34326","multi.service_calls":0,"queuetimems":27274,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:17:55,654 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646637175653","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:53416","multi.service_calls":0,"queuetimems":37369,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:18:02,280 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646636282278","responsesize":"16534","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 10 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.155:34278","multi.service_calls":0,"queuetimems":44098,"class":"HRegionServer","multi.mutations":10} 2022-03-07 07:18:16,201 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1316ms No GCs detected 2022-03-07 07:18:18,282 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646637198281","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:53416","multi.service_calls":0,"queuetimems":59997,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:18:19,481 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646637199480","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:34326","multi.service_calls":0,"queuetimems":61196,"class":"HRegionServer","multi.mutations":2} 2022-03-07 07:18:20,397 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646637200397","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:53416","multi.service_calls":0,"queuetimems":62113,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:18:20,674 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646637200674","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:53416","multi.service_calls":0,"queuetimems":62390,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:18:20,675 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646637200675","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:53416","multi.service_calls":0,"queuetimems":62391,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:18:37,608 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646636917607","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":600001,"client":"172.30.41.118:57576","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-07 07:18:49,220 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646637229220","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:34330","multi.service_calls":0,"queuetimems":8243,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=277239, hits=269129, hitRatio=97.07%, , cachingAccesses=269259, cachingHits=268820, cachingHitsRatio=99.84%, evictions=18589, evicted=0, evictedPerRun=0.0 2022-03-07 07:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 07:22:18,184 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646637438184","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:57668","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:22:18,284 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646637138283","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:57606","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 07:22:21,821 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646637441820","responsesize":"9917","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:57668","multi.service_calls":0,"queuetimems":3636,"class":"HRegionServer","multi.mutations":6} 2022-03-07 07:22:39,602 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":1,"starttimems":"1646636859600","responsesize":"6362","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:34308","multi.service_calls":0,"queuetimems":21318,"class":"HRegionServer","multi.mutations":3} 2022-03-07 07:22:55,654 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646637475654","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300000,"client":"172.30.41.118:57668","multi.service_calls":0,"queuetimems":33833,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:23:20,571 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646637500397","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300174,"client":"172.30.41.155:34344","multi.service_calls":0,"queuetimems":2112,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:23:20,587 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646637499481","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":301106,"client":"172.30.41.118:57668","multi.service_calls":0,"queuetimems":23827,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:23:20,675 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646637500674","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:53458","multi.service_calls":0,"queuetimems":1192,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:23:40,980 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646637220977","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":600003,"client":"172.30.41.118:57624","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 07:24:10,019 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1146ms No GCs detected 2022-03-07 07:26:27,998 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1294ms No GCs detected 2022-03-07 07:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=277952, hits=269842, hitRatio=97.08%, , cachingAccesses=269972, cachingHits=269533, cachingHitsRatio=99.84%, evictions=18619, evicted=0, evictedPerRun=0.0 2022-03-07 07:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 07:26:54,106 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-07 07:27:18,186 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646637738185","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300000,"client":"172.30.41.118:57726","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:27:18,286 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646637738285","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:34356","multi.service_calls":0,"queuetimems":100,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:27:25,815 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1614ms No GCs detected 2022-03-07 07:27:38,493 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1176ms No GCs detected 2022-03-07 07:27:39,602 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646637759602","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:57726","multi.service_calls":0,"queuetimems":21417,"class":"HRegionServer","multi.mutations":3} 2022-03-07 07:27:45,559 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646637465558","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:34342","multi.service_calls":0,"queuetimems":27374,"class":"HRegionServer","multi.mutations":4} 2022-03-07 07:28:02,281 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646637482280","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 6 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:57668","multi.service_calls":0,"queuetimems":40459,"class":"HRegionServer","multi.mutations":6} 2022-03-07 07:28:18,311 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646637498283","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600028,"client":"172.31.0.219:53458","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 07:28:20,588 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646637800588","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300000,"client":"172.30.41.118:57726","multi.service_calls":0,"queuetimems":10048,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:28:20,676 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646637500675","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:57668","multi.service_calls":0,"queuetimems":25020,"class":"HRegionServer","multi.mutations":4} 2022-03-07 07:28:20,676 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646637800676","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:34358","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:28:37,620 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646637517608","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600012,"client":"172.30.41.155:34344","multi.service_calls":0,"queuetimems":18126,"class":"HRegionServer","multi.mutations":3} 2022-03-07 07:28:49,229 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646637529220","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600009,"client":"172.31.0.219:53458","multi.service_calls":0,"queuetimems":30937,"class":"HRegionServer","multi.mutations":4} 2022-03-07 07:30:26,891 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1554ms No GCs detected 2022-03-07 07:30:34,279 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1736ms No GCs detected 2022-03-07 07:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=278742, hits=270632, hitRatio=97.09%, , cachingAccesses=270762, cachingHits=270323, cachingHitsRatio=99.84%, evictions=18649, evicted=0, evictedPerRun=0.0 2022-03-07 07:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-07 07:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 07:32:18,187 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646638038186","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:57792","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 07:32:21,822 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646637741821","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 8 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:53490","multi.service_calls":0,"queuetimems":3635,"class":"HRegionServer","multi.mutations":8} 2022-03-07 07:32:46,718 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646637166716","responsesize":"21490","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 13 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.118:57606","multi.service_calls":0,"queuetimems":28432,"class":"HRegionServer","multi.mutations":13} 2022-03-07 07:33:02,282 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646638082281","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:57798","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:33:20,651 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646638100588","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300063,"client":"172.30.41.118:57798","multi.service_calls":0,"queuetimems":18306,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:33:20,652 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646637800571","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600081,"client":"172.30.41.118:57726","multi.service_calls":0,"queuetimems":62286,"class":"HRegionServer","multi.mutations":2} 2022-03-07 07:33:20,994 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646638100676","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300317,"client":"172.30.41.118:57798","multi.service_calls":0,"queuetimems":18394,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=279470, hits=271360, hitRatio=97.10%, , cachingAccesses=271490, cachingHits=271051, cachingHitsRatio=99.84%, evictions=18679, evicted=0, evictedPerRun=0.0 2022-03-07 07:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 07:37:18,189 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646638338188","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:57858","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:37:18,287 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646638038287","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:34370","multi.service_calls":0,"queuetimems":101,"class":"HRegionServer","multi.mutations":2} 2022-03-07 07:37:39,604 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646638059602","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.155:34370","multi.service_calls":0,"queuetimems":21415,"class":"HRegionServer","multi.mutations":4} 2022-03-07 07:37:45,560 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646638065559","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 5 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:53504","multi.service_calls":0,"queuetimems":27372,"class":"HRegionServer","multi.mutations":5} 2022-03-07 07:37:46,718 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646638366718","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:57858","multi.service_calls":0,"queuetimems":24895,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:37:54,318 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1755ms No GCs detected 2022-03-07 07:38:20,670 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646638400669","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:57862","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:38:20,670 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646638400670","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:57862","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:38:21,537 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646638401536","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:57862","multi.service_calls":0,"queuetimems":867,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:38:40,982 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646637820981","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:34358","multi.service_calls":0,"queuetimems":20392,"class":"HRegionServer","multi.mutations":6} 2022-03-07 07:38:49,230 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646638129229","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":600001,"client":"172.30.41.118:57798","multi.service_calls":0,"queuetimems":28640,"class":"HRegionServer","multi.mutations":2} 2022-03-07 07:40:24,967 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2176ms No GCs detected 2022-03-07 07:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=280232, hits=272122, hitRatio=97.11%, , cachingAccesses=272252, cachingHits=271813, cachingHitsRatio=99.84%, evictions=18709, evicted=0, evictedPerRun=0.0 2022-03-07 07:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 07:42:18,288 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646638638288","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:57912","multi.service_calls":0,"queuetimems":98,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:42:39,605 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646638659604","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:34412","multi.service_calls":0,"queuetimems":21414,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:42:46,349 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646638666348","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:53538","multi.service_calls":0,"queuetimems":28158,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:42:46,719 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646638666718","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:34412","multi.service_calls":0,"queuetimems":28528,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:42:46,719 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] regionserver.RSRpcServices: Client tried to access missing scanner 770447983929072242 2022-03-07 07:42:57,128 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646637775654","responsesize":"21490","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 13 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1201473,"client":"172.30.41.118:57726","multi.service_calls":0,"queuetimems":37368,"class":"HRegionServer","multi.mutations":13} 2022-03-07 07:43:20,671 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646638700670","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:57926","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:43:20,678 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646638100676","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 7 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.31.0.219:53506","multi.service_calls":0,"queuetimems":18394,"class":"HRegionServer","multi.mutations":7} 2022-03-07 07:43:21,538 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646638701537","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:34414","multi.service_calls":0,"queuetimems":866,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:43:40,983 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646638720982","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:53540","multi.service_calls":0,"queuetimems":20311,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:43:49,231 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646638729230","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:53540","multi.service_calls":0,"queuetimems":28559,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:45:00,705 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1174ms No GCs detected 2022-03-07 07:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=281078, hits=272968, hitRatio=97.11%, , cachingAccesses=273098, cachingHits=272659, cachingHitsRatio=99.84%, evictions=18738, evicted=0, evictedPerRun=0.0 2022-03-07 07:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 07:47:18,191 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646638638190","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:57912","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-07 07:47:21,824 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646638341822","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 6 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:34394","multi.service_calls":0,"queuetimems":3634,"class":"HRegionServer","multi.mutations":6} 2022-03-07 07:47:46,350 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646638966349","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300001,"client":"172.30.41.118:57984","multi.service_calls":0,"queuetimems":6743,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:47:46,720 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646638966719","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:34430","multi.service_calls":0,"queuetimems":28429,"class":"HRegionServer","multi.mutations":2} 2022-03-07 07:48:02,283 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646638382282","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:53520","multi.service_calls":0,"queuetimems":44094,"class":"HRegionServer","multi.mutations":4} 2022-03-07 07:48:18,433 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646638098311","responsesize":"9926","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 6 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200122,"client":"172.30.41.155:34372","multi.service_calls":0,"queuetimems":16029,"class":"HRegionServer","multi.mutations":6} 2022-03-07 07:48:37,622 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646638117620","responsesize":"23142","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 14 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.118:57798","multi.service_calls":0,"queuetimems":17031,"class":"HRegionServer","multi.mutations":14} 2022-03-07 07:48:49,232 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646639029231","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:58006","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 07:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=281840, hits=273730, hitRatio=97.12%, , cachingAccesses=273860, cachingHits=273421, cachingHitsRatio=99.84%, evictions=18768, evicted=0, evictedPerRun=0.0 2022-03-07 07:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 07:52:18,290 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646638938289","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:57984","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 07:52:46,350 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646639266350","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:34446","multi.service_calls":0,"queuetimems":28158,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:52:57,129 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646638977128","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.31.0.219:53556","multi.service_calls":0,"queuetimems":38838,"class":"HRegionServer","multi.mutations":4} 2022-03-07 07:53:02,284 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646639282284","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:53590","multi.service_calls":0,"queuetimems":44091,"class":"HRegionServer","multi.mutations":2} 2022-03-07 07:53:20,672 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646639000671","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:34430","multi.service_calls":0,"queuetimems":41065,"class":"HRegionServer","multi.mutations":3} 2022-03-07 07:53:20,680 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646639000680","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.31.0.219:53566","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 07:53:41,046 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646639020983","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 4 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600063,"client":"172.31.0.219:53566","multi.service_calls":0,"queuetimems":20303,"class":"HRegionServer","multi.mutations":4} 2022-03-07 07:53:49,233 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646639329232","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:58072","multi.service_calls":0,"queuetimems":30798,"class":"HRegionServer","multi.mutations":2} 2022-03-07 07:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=282678, hits=274568, hitRatio=97.13%, , cachingAccesses=274698, cachingHits=274259, cachingHitsRatio=99.84%, evictions=18798, evicted=0, evictedPerRun=0.0 2022-03-07 07:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 07:57:18,193 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646639238192","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:58062","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":5} 2022-03-07 07:57:18,292 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646639538291","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300001,"client":"172.30.41.118:58114","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:57:23,524 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2787ms No GCs detected 2022-03-07 07:57:43,492 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-07 07:57:46,352 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646639566351","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:58114","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:58:02,285 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646639582284","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:34462","multi.service_calls":0,"queuetimems":15933,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:58:18,438 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646639298434","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600004,"client":"172.30.41.155:34450","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 07:58:20,676 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646639600673","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300003,"client":"172.31.0.219:53644","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:58:20,774 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646639600770","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300004,"client":"172.30.41.118:58126","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 07:58:21,540 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646639001538","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:34430","multi.service_calls":0,"queuetimems":63248,"class":"HRegionServer","multi.mutations":7} 2022-03-07 08:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=283504, hits=275394, hitRatio=97.14%, , cachingAccesses=275524, cachingHits=275085, cachingHitsRatio=99.84%, evictions=18828, evicted=0, evictedPerRun=0.0 2022-03-07 08:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 08:02:18,195 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646639838194","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:58178","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 08:02:18,293 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646639838292","responsesize":"8265","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 5 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:58178","multi.service_calls":0,"queuetimems":98,"class":"HRegionServer","multi.mutations":5} 2022-03-07 08:02:22,419 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646639241824","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 4 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900595,"client":"172.31.0.219:53590","multi.service_calls":0,"queuetimems":3632,"class":"HRegionServer","multi.mutations":4} 2022-03-07 08:02:39,607 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646638959605","responsesize":"21490","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 13 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.118:57984","multi.service_calls":0,"queuetimems":21316,"class":"HRegionServer","multi.mutations":13} 2022-03-07 08:02:46,352 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646639866352","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:53674","multi.service_calls":0,"queuetimems":28157,"class":"HRegionServer","multi.mutations":2} 2022-03-07 08:02:46,721 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646639266720","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 8 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:34446","multi.service_calls":0,"queuetimems":28528,"class":"HRegionServer","multi.mutations":8} 2022-03-07 08:02:57,130 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646639577129","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:34462","multi.service_calls":0,"queuetimems":38838,"class":"HRegionServer","multi.mutations":2} 2022-03-07 08:03:02,286 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646639882285","responsesize":"11569","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 7 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:58180","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":7} 2022-03-07 08:03:20,677 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646638700671","responsesize":"29753","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 18 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1500005,"client":"172.30.41.118:57926","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":18} 2022-03-07 08:03:20,708 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646639900708","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:58190","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 08:03:21,227 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646639901227","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300000,"client":"172.30.41.118:58192","multi.service_calls":0,"queuetimems":3,"class":"HRegionServer","multi.mutations":1} 2022-03-07 08:03:41,047 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646639621047","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 9 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.31.0.219:53644","multi.service_calls":0,"queuetimems":20373,"class":"HRegionServer","multi.mutations":9} 2022-03-07 08:03:49,234 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646639629233","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:53644","multi.service_calls":0,"queuetimems":8186,"class":"HRegionServer","multi.mutations":2} 2022-03-07 08:06:28,122 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1196ms No GCs detected 2022-03-07 08:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=284229, hits=276119, hitRatio=97.15%, , cachingAccesses=276249, cachingHits=275810, cachingHitsRatio=99.84%, evictions=18858, evicted=0, evictedPerRun=0.0 2022-03-07 08:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 08:06:54,301 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1959ms No GCs detected 2022-03-07 08:07:18,196 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646640138196","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:34496","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 08:07:18,294 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646640138293","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.30.41.155:34246","multi.service_calls":0,"queuetimems":10731,"class":"HRegionServer","multi.mutations":2} 2022-03-07 08:07:22,422 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646640142419","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.155:34496","multi.service_calls":0,"queuetimems":4223,"class":"HRegionServer","multi.mutations":1} 2022-03-07 08:07:46,722 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646640166721","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:58248","multi.service_calls":0,"queuetimems":28525,"class":"HRegionServer","multi.mutations":1} 2022-03-07 08:08:02,287 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646640182286","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:58248","multi.service_calls":0,"queuetimems":43728,"class":"HRegionServer","multi.mutations":1} 2022-03-07 08:08:18,494 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646639898438","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 5 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600056,"client":"172.30.41.155:34480","multi.service_calls":0,"queuetimems":16152,"class":"HRegionServer","multi.mutations":5} 2022-03-07 08:08:20,677 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646640200677","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:53694","multi.service_calls":0,"queuetimems":56061,"class":"HRegionServer","multi.mutations":1} 2022-03-07 08:08:20,709 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646640200708","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:53694","multi.service_calls":0,"queuetimems":62123,"class":"HRegionServer","multi.mutations":1} 2022-03-07 08:08:21,227 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646640201227","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:53694","multi.service_calls":0,"queuetimems":61687,"class":"HRegionServer","multi.mutations":1} 2022-03-07 08:08:21,541 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646639901540","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 5 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:34480","multi.service_calls":0,"queuetimems":19254,"class":"HRegionServer","multi.mutations":5} 2022-03-07 08:08:37,624 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646639317622","responsesize":"21490","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 13 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.31.0.219:53600","multi.service_calls":0,"queuetimems":19187,"class":"HRegionServer","multi.mutations":13} 2022-03-07 08:08:41,049 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646640221048","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:58254","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 08:08:45,827 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-07 08:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=284950, hits=276840, hitRatio=97.15%, , cachingAccesses=276970, cachingHits=276531, cachingHitsRatio=99.84%, evictions=18888, evicted=0, evictedPerRun=0.0 2022-03-07 08:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 08:12:18,295 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646640438295","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:34534","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 08:12:22,422 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646640442422","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:58300","multi.service_calls":0,"queuetimems":4127,"class":"HRegionServer","multi.mutations":1} 2022-03-07 08:12:39,608 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646640159607","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":600001,"client":"172.30.41.118:58248","multi.service_calls":0,"queuetimems":21411,"class":"HRegionServer","multi.mutations":2} 2022-03-07 08:13:18,494 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646640498494","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:53710","multi.service_calls":0,"queuetimems":16206,"class":"HRegionServer","multi.mutations":2} 2022-03-07 08:13:20,709 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646640500709","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:53710","multi.service_calls":0,"queuetimems":18421,"class":"HRegionServer","multi.mutations":1} 2022-03-07 08:13:21,228 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646640501228","responsesize":"6613","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:58304","multi.service_calls":0,"queuetimems":550,"class":"HRegionServer","multi.mutations":4} 2022-03-07 08:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=285772, hits=277662, hitRatio=97.16%, , cachingAccesses=277792, cachingHits=277353, cachingHitsRatio=99.84%, evictions=18918, evicted=0, evictedPerRun=0.0 2022-03-07 08:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 08:17:18,198 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646640438197","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:58300","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 08:17:18,297 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646640738296","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:58346","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 08:17:22,423 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646640742422","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:58346","multi.service_calls":0,"queuetimems":4126,"class":"HRegionServer","multi.mutations":2} 2022-03-07 08:17:39,609 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646640759609","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:53724","multi.service_calls":0,"queuetimems":21312,"class":"HRegionServer","multi.mutations":1} 2022-03-07 08:17:46,354 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646640166352","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.31.0.219:53694","multi.service_calls":0,"queuetimems":28156,"class":"HRegionServer","multi.mutations":3} 2022-03-07 08:17:57,131 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646640177130","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 5 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:53694","multi.service_calls":0,"queuetimems":38934,"class":"HRegionServer","multi.mutations":5} 2022-03-07 08:18:02,288 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646640482287","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:34540","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 08:18:18,495 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646640798494","responsesize":"8265","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:53724","multi.service_calls":0,"queuetimems":60197,"class":"HRegionServer","multi.mutations":5} 2022-03-07 08:18:20,679 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646640500678","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:34540","multi.service_calls":0,"queuetimems":18391,"class":"HRegionServer","multi.mutations":4} 2022-03-07 08:18:20,710 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646640800709","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:58346","multi.service_calls":0,"queuetimems":33056,"class":"HRegionServer","multi.mutations":1} 2022-03-07 08:18:21,229 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646640801228","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:53724","multi.service_calls":0,"queuetimems":33426,"class":"HRegionServer","multi.mutations":1} 2022-03-07 08:18:21,542 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646640501541","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 5 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:58304","multi.service_calls":0,"queuetimems":19251,"class":"HRegionServer","multi.mutations":5} 2022-03-07 08:18:37,625 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646640517624","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:58304","multi.service_calls":0,"queuetimems":16946,"class":"HRegionServer","multi.mutations":5} 2022-03-07 08:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=286538, hits=278428, hitRatio=97.17%, , cachingAccesses=278558, cachingHits=278119, cachingHitsRatio=99.84%, evictions=18948, evicted=0, evictedPerRun=0.0 2022-03-07 08:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 08:22:18,200 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646641038199","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:58400","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 08:22:31,295 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1506ms No GCs detected 2022-03-07 08:22:39,610 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646641059609","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:58400","multi.service_calls":0,"queuetimems":21410,"class":"HRegionServer","multi.mutations":3} 2022-03-07 08:22:46,354 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646641066354","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:34606","multi.service_calls":0,"queuetimems":28155,"class":"HRegionServer","multi.mutations":2} 2022-03-07 08:22:57,132 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646641077131","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:58400","multi.service_calls":0,"queuetimems":38833,"class":"HRegionServer","multi.mutations":1} 2022-03-07 08:23:20,681 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646641100680","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:34608","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 08:23:20,711 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646641100710","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:58414","multi.service_calls":0,"queuetimems":30,"class":"HRegionServer","multi.mutations":1} 2022-03-07 08:23:41,050 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646640521049","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 5 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:58304","multi.service_calls":0,"queuetimems":38759,"class":"HRegionServer","multi.mutations":5} 2022-03-07 08:23:49,236 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646640229234","responsesize":"24794","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 15 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.118:58254","multi.service_calls":0,"queuetimems":8186,"class":"HRegionServer","multi.mutations":15} 2022-03-07 08:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=287276, hits=279166, hitRatio=97.18%, , cachingAccesses=279296, cachingHits=278857, cachingHitsRatio=99.84%, evictions=18978, evicted=0, evictedPerRun=0.0 2022-03-07 08:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 08:27:18,201 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646641338201","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:34624","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 08:27:18,298 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646641038297","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:53758","multi.service_calls":0,"queuetimems":97,"class":"HRegionServer","multi.mutations":4} 2022-03-07 08:27:22,425 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646641042423","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.31.0.219:53758","multi.service_calls":0,"queuetimems":4223,"class":"HRegionServer","multi.mutations":6} 2022-03-07 08:27:46,355 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646641366354","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:53798","multi.service_calls":0,"queuetimems":28153,"class":"HRegionServer","multi.mutations":1} 2022-03-07 08:27:46,723 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646640466722","responsesize":"16534","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 10 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200001,"client":"172.30.41.155:34534","multi.service_calls":0,"queuetimems":28427,"class":"HRegionServer","multi.mutations":10} 2022-03-07 08:27:57,132 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646641377132","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:53798","multi.service_calls":0,"queuetimems":38931,"class":"HRegionServer","multi.mutations":1} 2022-03-07 08:28:02,289 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646641082288","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:34606","multi.service_calls":0,"queuetimems":44089,"class":"HRegionServer","multi.mutations":3} 2022-03-07 08:28:20,683 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646641400681","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.30.41.118:58242","multi.service_calls":0,"queuetimems":12886,"class":"HRegionServer","multi.mutations":2} 2022-03-07 08:28:20,711 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646641400711","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:53798","multi.service_calls":0,"queuetimems":62510,"class":"HRegionServer","multi.mutations":1} 2022-03-07 08:28:21,230 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646641101229","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:34608","multi.service_calls":0,"queuetimems":549,"class":"HRegionServer","multi.mutations":3} 2022-03-07 08:28:37,626 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646641117625","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 5 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:58414","multi.service_calls":0,"queuetimems":16945,"class":"HRegionServer","multi.mutations":5} 2022-03-07 08:28:41,051 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646641421050","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:53798","multi.service_calls":0,"queuetimems":24573,"class":"HRegionServer","multi.mutations":1} 2022-03-07 08:28:49,237 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646641429236","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:53798","multi.service_calls":0,"queuetimems":32759,"class":"HRegionServer","multi.mutations":1} 2022-03-07 08:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=288078, hits=279968, hitRatio=97.18%, , cachingAccesses=280098, cachingHits=279659, cachingHitsRatio=99.84%, evictions=19008, evicted=0, evictedPerRun=0.0 2022-03-07 08:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-07 08:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 08:32:18,203 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646641638202","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:58516","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 08:32:18,299 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646641638299","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:34636","multi.service_calls":0,"queuetimems":97,"class":"HRegionServer","multi.mutations":1} 2022-03-07 08:32:39,611 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646641359610","responsesize":"11572","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 7 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:58460","multi.service_calls":0,"queuetimems":21409,"class":"HRegionServer","multi.mutations":7} 2022-03-07 08:32:46,724 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646641666723","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:34636","multi.service_calls":0,"queuetimems":28521,"class":"HRegionServer","multi.mutations":1} 2022-03-07 08:33:02,290 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646641682289","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:53828","multi.service_calls":0,"queuetimems":43990,"class":"HRegionServer","multi.mutations":1} 2022-03-07 08:33:18,496 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646641098495","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 5 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:53758","multi.service_calls":0,"queuetimems":60197,"class":"HRegionServer","multi.mutations":5} 2022-03-07 08:33:20,684 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646641700683","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:58532","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 08:33:21,230 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646641701230","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:53830","multi.service_calls":0,"queuetimems":546,"class":"HRegionServer","multi.mutations":1} 2022-03-07 08:33:37,630 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646641717626","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300004,"client":"172.31.0.219:53830","multi.service_calls":0,"queuetimems":16942,"class":"HRegionServer","multi.mutations":2} 2022-03-07 08:33:41,051 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646641721051","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:34638","multi.service_calls":0,"queuetimems":20368,"class":"HRegionServer","multi.mutations":1} 2022-03-07 08:33:49,237 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646641729237","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:34638","multi.service_calls":0,"queuetimems":28554,"class":"HRegionServer","multi.mutations":2} 2022-03-07 08:35:44,045 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1351ms No GCs detected 2022-03-07 08:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=288768, hits=280658, hitRatio=97.19%, , cachingAccesses=280788, cachingHits=280349, cachingHitsRatio=99.84%, evictions=19038, evicted=0, evictedPerRun=0.0 2022-03-07 08:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 08:37:18,204 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646641938204","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:58598","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 08:37:18,300 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646641938300","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:34656","multi.service_calls":0,"queuetimems":96,"class":"HRegionServer","multi.mutations":1} 2022-03-07 08:37:46,356 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646641666355","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 6 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:34636","multi.service_calls":0,"queuetimems":28153,"class":"HRegionServer","multi.mutations":6} 2022-03-07 08:37:57,134 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646641677133","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 9 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.31.0.219:53828","multi.service_calls":0,"queuetimems":38930,"class":"HRegionServer","multi.mutations":9} 2022-03-07 08:38:02,291 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646641982290","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:58598","multi.service_calls":0,"queuetimems":43990,"class":"HRegionServer","multi.mutations":1} 2022-03-07 08:38:18,497 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646641998497","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:58608","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 08:38:20,685 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646642000685","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:34658","multi.service_calls":0,"queuetimems":2188,"class":"HRegionServer","multi.mutations":3} 2022-03-07 08:38:21,544 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646641101542","responsesize":"23142","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 14 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.31.0.219:53768","multi.service_calls":0,"queuetimems":861,"class":"HRegionServer","multi.mutations":14} 2022-03-07 08:38:37,630 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646642017630","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:53850","multi.service_calls":0,"queuetimems":19133,"class":"HRegionServer","multi.mutations":1} 2022-03-07 08:38:41,052 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646642021052","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:58608","multi.service_calls":0,"queuetimems":20367,"class":"HRegionServer","multi.mutations":1} 2022-03-07 08:38:49,238 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646642029237","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:53850","multi.service_calls":0,"queuetimems":28552,"class":"HRegionServer","multi.mutations":1} 2022-03-07 08:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=289547, hits=281437, hitRatio=97.20%, , cachingAccesses=281567, cachingHits=281128, cachingHitsRatio=99.84%, evictions=19068, evicted=0, evictedPerRun=0.0 2022-03-07 08:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 08:42:18,301 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646642238301","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:53866","multi.service_calls":0,"queuetimems":96,"class":"HRegionServer","multi.mutations":1} 2022-03-07 08:42:46,356 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646642266356","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:53866","multi.service_calls":0,"queuetimems":28150,"class":"HRegionServer","multi.mutations":1} 2022-03-07 08:42:46,725 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646641966724","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 6 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:34656","multi.service_calls":0,"queuetimems":28520,"class":"HRegionServer","multi.mutations":6} 2022-03-07 08:43:02,291 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646642282291","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:53866","multi.service_calls":0,"queuetimems":44085,"class":"HRegionServer","multi.mutations":1} 2022-03-07 08:43:18,499 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646642298498","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:58682","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 08:43:20,713 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646641700712","responsesize":"21487","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 13 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.30.41.118:58532","multi.service_calls":0,"queuetimems":29,"class":"HRegionServer","multi.mutations":13} 2022-03-07 08:43:49,238 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646642329238","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:58682","multi.service_calls":0,"queuetimems":28552,"class":"HRegionServer","multi.mutations":1} 2022-03-07 08:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=290240, hits=282130, hitRatio=97.21%, , cachingAccesses=282260, cachingHits=281821, cachingHitsRatio=99.84%, evictions=19098, evicted=0, evictedPerRun=0.0 2022-03-07 08:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 08:47:18,206 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646642238205","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 5 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:58668","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":5} 2022-03-07 08:47:22,428 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646641642425","responsesize":"23142","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 14 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.118:58516","multi.service_calls":0,"queuetimems":4223,"class":"HRegionServer","multi.mutations":14} 2022-03-07 08:47:39,612 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646641959611","responsesize":"18183","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 11 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.30.41.118:58598","multi.service_calls":0,"queuetimems":21407,"class":"HRegionServer","multi.mutations":11} 2022-03-07 08:48:20,714 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646642600713","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:34694","multi.service_calls":0,"queuetimems":2213,"class":"HRegionServer","multi.mutations":2} 2022-03-07 08:48:21,232 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646642001230","responsesize":"18183","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 11 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900002,"client":"172.30.41.118:58608","multi.service_calls":0,"queuetimems":2733,"class":"HRegionServer","multi.mutations":11} 2022-03-07 08:48:21,545 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646642301544","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:34676","multi.service_calls":0,"queuetimems":3045,"class":"HRegionServer","multi.mutations":5} 2022-03-07 08:48:37,631 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646642317630","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:34676","multi.service_calls":0,"queuetimems":19132,"class":"HRegionServer","multi.mutations":4} 2022-03-07 08:48:41,058 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646642321052","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 8 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600006,"client":"172.31.0.219:53868","multi.service_calls":0,"queuetimems":22553,"class":"HRegionServer","multi.mutations":8} 2022-03-07 08:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=291000, hits=282890, hitRatio=97.21%, , cachingAccesses=283020, cachingHits=282581, cachingHitsRatio=99.84%, evictions=19128, evicted=0, evictedPerRun=0.0 2022-03-07 08:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 08:52:18,303 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646642538302","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:58744","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 08:52:46,358 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646642566357","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 4 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:53884","multi.service_calls":0,"queuetimems":28054,"class":"HRegionServer","multi.mutations":4} 2022-03-07 08:52:57,135 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646642277134","responsesize":"16531","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 10 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.30.41.118:58668","multi.service_calls":0,"queuetimems":38929,"class":"HRegionServer","multi.mutations":10} 2022-03-07 08:53:02,292 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646642582291","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 6 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:58744","multi.service_calls":0,"queuetimems":15934,"class":"HRegionServer","multi.mutations":6} 2022-03-07 08:53:18,500 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646642598500","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:34694","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 08:53:20,688 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646642300686","responsesize":"19835","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 12 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900002,"client":"172.30.41.118:58682","multi.service_calls":0,"queuetimems":2188,"class":"HRegionServer","multi.mutations":12} 2022-03-07 08:53:21,232 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646642901232","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:58810","multi.service_calls":0,"queuetimems":58804,"class":"HRegionServer","multi.mutations":1} 2022-03-07 08:53:21,546 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646642901546","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:58826","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 08:53:37,632 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646642917632","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:58826","multi.service_calls":0,"queuetimems":16086,"class":"HRegionServer","multi.mutations":1} 2022-03-07 08:53:49,240 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646642629239","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:58754","multi.service_calls":0,"queuetimems":30739,"class":"HRegionServer","multi.mutations":3} 2022-03-07 08:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=291695, hits=283585, hitRatio=97.22%, , cachingAccesses=283715, cachingHits=283276, cachingHitsRatio=99.85%, evictions=19158, evicted=0, evictedPerRun=0.0 2022-03-07 08:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 08:57:18,208 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646642838207","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 5 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:34704","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":5} 2022-03-07 08:57:18,304 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646643138304","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:58878","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 08:57:46,359 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646643166359","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:34720","multi.service_calls":0,"queuetimems":28055,"class":"HRegionServer","multi.mutations":2} 2022-03-07 08:57:46,726 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646642566725","responsesize":"16531","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 10 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.30.41.118:58744","multi.service_calls":0,"queuetimems":28423,"class":"HRegionServer","multi.mutations":10} 2022-03-07 08:57:57,136 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646643177135","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:53920","multi.service_calls":0,"queuetimems":38831,"class":"HRegionServer","multi.mutations":1} 2022-03-07 08:58:18,502 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646643198501","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:34722","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 08:58:20,690 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646643200688","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.118:58878","multi.service_calls":0,"queuetimems":34329,"class":"HRegionServer","multi.mutations":1} 2022-03-07 08:58:21,233 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646643201232","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:53922","multi.service_calls":0,"queuetimems":2730,"class":"HRegionServer","multi.mutations":1} 2022-03-07 08:58:41,059 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646642921058","responsesize":"11572","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:53904","multi.service_calls":0,"queuetimems":19512,"class":"HRegionServer","multi.mutations":7} 2022-03-07 08:58:49,241 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646643229240","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:34722","multi.service_calls":0,"queuetimems":30739,"class":"HRegionServer","multi.mutations":2} 2022-03-07 09:01:44,382 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=292472, hits=284362, hitRatio=97.23%, , cachingAccesses=284492, cachingHits=284053, cachingHitsRatio=99.85%, evictions=19188, evicted=0, evictedPerRun=0.0 2022-03-07 09:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 09:02:18,210 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646643438209","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:34732","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 09:02:22,430 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646642842428","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 6 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":900002,"client":"172.30.41.118:58810","multi.service_calls":0,"queuetimems":4221,"class":"HRegionServer","multi.mutations":6} 2022-03-07 09:02:39,613 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646642859612","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:58810","multi.service_calls":0,"queuetimems":21405,"class":"HRegionServer","multi.mutations":3} 2022-03-07 09:02:46,727 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646643466726","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:53936","multi.service_calls":0,"queuetimems":28516,"class":"HRegionServer","multi.mutations":2} 2022-03-07 09:03:02,293 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646643182292","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:58878","multi.service_calls":0,"queuetimems":43988,"class":"HRegionServer","multi.mutations":4} 2022-03-07 09:03:18,502 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646643498502","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:58960","multi.service_calls":0,"queuetimems":60197,"class":"HRegionServer","multi.mutations":1} 2022-03-07 09:03:20,715 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646642900714","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 5 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:53902","multi.service_calls":0,"queuetimems":62506,"class":"HRegionServer","multi.mutations":5} 2022-03-07 09:03:21,234 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646643501233","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300000,"client":"172.30.41.118:58960","multi.service_calls":0,"queuetimems":39394,"class":"HRegionServer","multi.mutations":1} 2022-03-07 09:03:37,634 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646643217633","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:58898","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 09:03:41,060 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646643521059","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:53938","multi.service_calls":0,"queuetimems":20368,"class":"HRegionServer","multi.mutations":1} 2022-03-07 09:03:49,241 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646643529241","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:53938","multi.service_calls":0,"queuetimems":28550,"class":"HRegionServer","multi.mutations":1} 2022-03-07 09:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=293233, hits=285123, hitRatio=97.23%, , cachingAccesses=285253, cachingHits=284814, cachingHitsRatio=99.85%, evictions=19218, evicted=0, evictedPerRun=0.0 2022-03-07 09:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 09:07:18,306 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646643438305","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:58960","multi.service_calls":0,"queuetimems":96,"class":"HRegionServer","multi.mutations":4} 2022-03-07 09:07:22,432 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646643742431","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:34748","multi.service_calls":0,"queuetimems":4220,"class":"HRegionServer","multi.mutations":1} 2022-03-07 09:07:39,614 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646643759613","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:34748","multi.service_calls":0,"queuetimems":21402,"class":"HRegionServer","multi.mutations":1} 2022-03-07 09:07:46,727 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646643766727","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:59034","multi.service_calls":0,"queuetimems":28486,"class":"HRegionServer","multi.mutations":2} 2022-03-07 09:07:57,137 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646643477136","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 9 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:58960","multi.service_calls":0,"queuetimems":38831,"class":"HRegionServer","multi.mutations":9} 2022-03-07 09:08:02,294 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646643782293","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:59028","multi.service_calls":0,"queuetimems":39862,"class":"HRegionServer","multi.mutations":1} 2022-03-07 09:08:06,230 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1041ms No GCs detected 2022-03-07 09:08:08,792 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2061ms No GCs detected 2022-03-07 09:08:20,692 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646643500691","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:34734","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-07 09:08:21,234 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646643801234","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:59046","multi.service_calls":0,"queuetimems":488,"class":"HRegionServer","multi.mutations":2} 2022-03-07 09:08:21,548 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646643201546","responsesize":"21487","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 13 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900002,"client":"172.30.41.118:58878","multi.service_calls":0,"queuetimems":35187,"class":"HRegionServer","multi.mutations":13} 2022-03-07 09:08:24,964 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-07 09:08:37,635 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646643817635","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:59050","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 09:08:49,243 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646643829241","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.31.0.219:53956","multi.service_calls":0,"queuetimems":8180,"class":"HRegionServer","multi.mutations":1} 2022-03-07 09:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=293984, hits=285874, hitRatio=97.24%, , cachingAccesses=286004, cachingHits=285565, cachingHitsRatio=99.85%, evictions=19248, evicted=0, evictedPerRun=0.0 2022-03-07 09:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 09:12:46,360 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646643466359","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:58960","multi.service_calls":0,"queuetimems":28150,"class":"HRegionServer","multi.mutations":4} 2022-03-07 09:13:18,503 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646643798502","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 8 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:59028","multi.service_calls":0,"queuetimems":56070,"class":"HRegionServer","multi.mutations":8} 2022-03-07 09:13:20,717 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646643800716","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:53956","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-07 09:13:37,637 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646644117636","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:34768","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 09:13:41,061 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646643821060","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:53956","multi.service_calls":0,"queuetimems":20344,"class":"HRegionServer","multi.mutations":2} 2022-03-07 09:13:49,246 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646644129245","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:53986","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 09:14:39,518 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2174ms No GCs detected 2022-03-07 09:14:57,614 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-07 09:15:08,813 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1992ms No GCs detected 2022-03-07 09:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=294797, hits=286687, hitRatio=97.25%, , cachingAccesses=286817, cachingHits=286378, cachingHitsRatio=99.85%, evictions=19278, evicted=0, evictedPerRun=0.0 2022-03-07 09:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 09:17:18,212 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646643738211","responsesize":"14879","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 9 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:59028","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":9} 2022-03-07 09:17:18,307 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646644038307","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:59106","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-07 09:17:22,433 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646644042433","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:34766","multi.service_calls":0,"queuetimems":4126,"class":"HRegionServer","multi.mutations":2} 2022-03-07 09:17:39,615 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646644059614","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:59106","multi.service_calls":0,"queuetimems":21307,"class":"HRegionServer","multi.mutations":3} 2022-03-07 09:17:46,728 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646644066727","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:34766","multi.service_calls":0,"queuetimems":28420,"class":"HRegionServer","multi.mutations":4} 2022-03-07 09:17:57,138 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646644077137","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:53980","multi.service_calls":0,"queuetimems":38830,"class":"HRegionServer","multi.mutations":3} 2022-03-07 09:18:02,295 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646644082294","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:59106","multi.service_calls":0,"queuetimems":22679,"class":"HRegionServer","multi.mutations":2} 2022-03-07 09:18:20,694 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646644100692","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.31.0.219:53980","multi.service_calls":0,"queuetimems":62385,"class":"HRegionServer","multi.mutations":2} 2022-03-07 09:18:20,718 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646644400717","responsesize":"6613","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300001,"client":"172.30.41.118:59170","multi.service_calls":0,"queuetimems":34355,"class":"HRegionServer","multi.mutations":4} 2022-03-07 09:18:21,549 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646644101548","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 5 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:34766","multi.service_calls":0,"queuetimems":41933,"class":"HRegionServer","multi.mutations":5} 2022-03-07 09:18:49,247 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646644429246","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:54022","multi.service_calls":0,"queuetimems":7614,"class":"HRegionServer","multi.mutations":1} 2022-03-07 09:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=295533, hits=287423, hitRatio=97.26%, , cachingAccesses=287553, cachingHits=287114, cachingHitsRatio=99.85%, evictions=19308, evicted=0, evictedPerRun=0.0 2022-03-07 09:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 09:22:18,214 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646644638213","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:59222","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 09:22:39,616 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646644659615","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:34802","multi.service_calls":0,"queuetimems":21402,"class":"HRegionServer","multi.mutations":2} 2022-03-07 09:22:46,362 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646644366361","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:59170","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-07 09:22:57,139 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646644677139","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:59226","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 09:23:02,528 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646644682524","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300003,"client":"172.30.41.118:59226","multi.service_calls":0,"queuetimems":5385,"class":"HRegionServer","multi.mutations":1} 2022-03-07 09:23:20,695 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646644700694","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:59226","multi.service_calls":0,"queuetimems":23555,"class":"HRegionServer","multi.mutations":2} 2022-03-07 09:23:20,718 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646644700718","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:34804","multi.service_calls":0,"queuetimems":23579,"class":"HRegionServer","multi.mutations":1} 2022-03-07 09:23:21,236 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646644101234","responsesize":"16531","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 10 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.30.41.118:59106","multi.service_calls":0,"queuetimems":41619,"class":"HRegionServer","multi.mutations":10} 2022-03-07 09:23:21,549 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646644701549","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:34804","multi.service_calls":0,"queuetimems":24410,"class":"HRegionServer","multi.mutations":1} 2022-03-07 09:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=296173, hits=288063, hitRatio=97.26%, , cachingAccesses=288193, cachingHits=287754, cachingHitsRatio=99.85%, evictions=19338, evicted=0, evictedPerRun=0.0 2022-03-07 09:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 09:27:22,436 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646644642433","responsesize":"16528","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 10 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":600003,"client":"172.31.0.219:54048","multi.service_calls":0,"queuetimems":4219,"class":"HRegionServer","multi.mutations":10} 2022-03-07 09:27:39,616 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646644959616","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:34822","multi.service_calls":0,"queuetimems":21401,"class":"HRegionServer","multi.mutations":1} 2022-03-07 09:27:46,729 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646644666728","responsesize":"16528","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 10 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":600001,"client":"172.30.41.118:59222","multi.service_calls":0,"queuetimems":28420,"class":"HRegionServer","multi.mutations":10} 2022-03-07 09:27:57,140 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646644977140","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:34822","multi.service_calls":0,"queuetimems":38925,"class":"HRegionServer","multi.mutations":3} 2022-03-07 09:28:18,505 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646644398503","responsesize":"18183","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 11 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900002,"client":"172.30.41.118:59170","multi.service_calls":0,"queuetimems":32142,"class":"HRegionServer","multi.mutations":11} 2022-03-07 09:28:20,697 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646645000695","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.118:59292","multi.service_calls":0,"queuetimems":23555,"class":"HRegionServer","multi.mutations":1} 2022-03-07 09:28:20,719 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646645000719","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:54066","multi.service_calls":0,"queuetimems":23,"class":"HRegionServer","multi.mutations":1} 2022-03-07 09:28:21,236 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646645001236","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:59300","multi.service_calls":0,"queuetimems":487,"class":"HRegionServer","multi.mutations":2} 2022-03-07 09:28:37,639 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646644417638","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:34786","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-07 09:28:41,063 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646644421062","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 6 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:34786","multi.service_calls":0,"queuetimems":3424,"class":"HRegionServer","multi.mutations":6} 2022-03-07 09:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=296872, hits=288762, hitRatio=97.27%, , cachingAccesses=288892, cachingHits=288453, cachingHitsRatio=99.85%, evictions=19368, evicted=0, evictedPerRun=0.0 2022-03-07 09:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-07 09:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 09:32:18,216 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646644938215","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:59292","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 09:32:18,309 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646644638308","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:59222","multi.service_calls":0,"queuetimems":95,"class":"HRegionServer","multi.mutations":7} 2022-03-07 09:32:22,437 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646645242437","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:59344","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 09:32:39,617 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646645259617","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:54086","multi.service_calls":0,"queuetimems":17180,"class":"HRegionServer","multi.mutations":1} 2022-03-07 09:32:46,363 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646644966362","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:59292","multi.service_calls":0,"queuetimems":28147,"class":"HRegionServer","multi.mutations":3} 2022-03-07 09:32:46,730 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646645266729","responsesize":"9917","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:59344","multi.service_calls":0,"queuetimems":24292,"class":"HRegionServer","multi.mutations":6} 2022-03-07 09:32:57,141 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646645277140","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:34842","multi.service_calls":0,"queuetimems":34703,"class":"HRegionServer","multi.mutations":1} 2022-03-07 09:33:21,237 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646645301236","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:34844","multi.service_calls":0,"queuetimems":538,"class":"HRegionServer","multi.mutations":1} 2022-03-07 09:33:21,550 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646645001549","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:34822","multi.service_calls":0,"queuetimems":24409,"class":"HRegionServer","multi.mutations":2} 2022-03-07 09:33:41,064 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646645321063","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:54090","multi.service_calls":0,"queuetimems":22557,"class":"HRegionServer","multi.mutations":1} 2022-03-07 09:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=297630, hits=289520, hitRatio=97.28%, , cachingAccesses=289650, cachingHits=289211, cachingHitsRatio=99.85%, evictions=19398, evicted=0, evictedPerRun=0.0 2022-03-07 09:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 09:37:22,439 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646645542437","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.31.0.219:54116","multi.service_calls":0,"queuetimems":4219,"class":"HRegionServer","multi.mutations":1} 2022-03-07 09:37:46,364 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646645566363","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:59402","multi.service_calls":0,"queuetimems":28146,"class":"HRegionServer","multi.mutations":3} 2022-03-07 09:37:57,142 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646645577142","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:59404","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 09:38:02,529 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646644982528","responsesize":"21487","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 13 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.30.41.118:59292","multi.service_calls":0,"queuetimems":5388,"class":"HRegionServer","multi.mutations":13} 2022-03-07 09:38:18,506 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646645298505","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:59360","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 09:38:20,698 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646645300698","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:59360","multi.service_calls":0,"queuetimems":2192,"class":"HRegionServer","multi.mutations":3} 2022-03-07 09:38:21,237 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646645601237","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:54116","multi.service_calls":0,"queuetimems":62927,"class":"HRegionServer","multi.mutations":1} 2022-03-07 09:38:37,640 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646645317639","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:54090","multi.service_calls":0,"queuetimems":16941,"class":"HRegionServer","multi.mutations":2} 2022-03-07 09:38:41,065 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646645621065","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:59410","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 09:38:49,249 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646644729247","responsesize":"16534","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 10 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200001,"client":"172.31.0.219:54050","multi.service_calls":0,"queuetimems":52108,"class":"HRegionServer","multi.mutations":10} 2022-03-07 09:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=298397, hits=290287, hitRatio=97.28%, , cachingAccesses=290417, cachingHits=289978, cachingHitsRatio=99.85%, evictions=19428, evicted=0, evictedPerRun=0.0 2022-03-07 09:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 09:41:53,581 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1193ms No GCs detected 2022-03-07 09:42:18,226 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646645538217","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600009,"client":"172.30.41.155:34860","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-07 09:42:18,311 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646645538310","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:54116","multi.service_calls":0,"queuetimems":92,"class":"HRegionServer","multi.mutations":2} 2022-03-07 09:42:39,619 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646645559618","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:34860","multi.service_calls":0,"queuetimems":21401,"class":"HRegionServer","multi.mutations":5} 2022-03-07 09:43:20,701 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646645900699","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300001,"client":"172.30.41.118:59450","multi.service_calls":0,"queuetimems":34334,"class":"HRegionServer","multi.mutations":2} 2022-03-07 09:43:20,721 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646645300719","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 7 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.155:34844","multi.service_calls":0,"queuetimems":2213,"class":"HRegionServer","multi.mutations":7} 2022-03-07 09:43:21,551 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646645601550","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:34862","multi.service_calls":0,"queuetimems":24408,"class":"HRegionServer","multi.mutations":5} 2022-03-07 09:43:37,641 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646645917641","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:59456","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 09:43:41,066 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646645921066","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:34886","multi.service_calls":0,"queuetimems":3425,"class":"HRegionServer","multi.mutations":1} 2022-03-07 09:43:49,249 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646645929249","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:34886","multi.service_calls":0,"queuetimems":11608,"class":"HRegionServer","multi.mutations":2} 2022-03-07 09:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=299231, hits=291121, hitRatio=97.29%, , cachingAccesses=291251, cachingHits=290812, cachingHitsRatio=99.85%, evictions=19458, evicted=0, evictedPerRun=0.0 2022-03-07 09:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 09:47:18,228 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646646138227","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:34916","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 09:47:22,442 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646645842441","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:59450","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 09:47:39,619 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646646159619","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:54174","multi.service_calls":0,"queuetimems":21391,"class":"HRegionServer","multi.mutations":1} 2022-03-07 09:47:46,365 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646645866364","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:34878","multi.service_calls":0,"queuetimems":23923,"class":"HRegionServer","multi.mutations":4} 2022-03-07 09:47:46,732 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646645566730","responsesize":"14879","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 9 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:59402","multi.service_calls":0,"queuetimems":28420,"class":"HRegionServer","multi.mutations":9} 2022-03-07 09:48:18,507 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646645898507","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:59450","multi.service_calls":0,"queuetimems":32142,"class":"HRegionServer","multi.mutations":4} 2022-03-07 09:48:20,702 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646646200702","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:34920","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 09:48:20,722 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646646200721","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:59502","multi.service_calls":0,"queuetimems":19,"class":"HRegionServer","multi.mutations":1} 2022-03-07 09:48:21,238 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646645901237","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:34878","multi.service_calls":0,"queuetimems":34872,"class":"HRegionServer","multi.mutations":4} 2022-03-07 09:48:41,067 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646646221066","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:34920","multi.service_calls":0,"queuetimems":20344,"class":"HRegionServer","multi.mutations":1} 2022-03-07 09:48:49,250 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646646229249","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:34920","multi.service_calls":0,"queuetimems":28527,"class":"HRegionServer","multi.mutations":1} 2022-03-07 09:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=300069, hits=291959, hitRatio=97.30%, , cachingAccesses=292089, cachingHits=291650, cachingHitsRatio=99.85%, evictions=19488, evicted=0, evictedPerRun=0.0 2022-03-07 09:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 09:52:18,229 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646646438228","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.30.41.118:59346","multi.service_calls":0,"queuetimems":9508,"class":"HRegionServer","multi.mutations":2} 2022-03-07 09:52:22,444 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646646442443","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:59552","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 09:52:39,620 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646646459620","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:59552","multi.service_calls":0,"queuetimems":17177,"class":"HRegionServer","multi.mutations":1} 2022-03-07 09:52:46,732 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646646466732","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:34954","multi.service_calls":0,"queuetimems":24289,"class":"HRegionServer","multi.mutations":1} 2022-03-07 09:52:57,144 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646645877142","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.155:34878","multi.service_calls":0,"queuetimems":34701,"class":"HRegionServer","multi.mutations":3} 2022-03-07 09:53:02,531 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646645882529","responsesize":"18183","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 11 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900002,"client":"172.30.41.118:59450","multi.service_calls":0,"queuetimems":40088,"class":"HRegionServer","multi.mutations":11} 2022-03-07 09:53:21,239 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646646501238","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:54214","multi.service_calls":0,"queuetimems":2729,"class":"HRegionServer","multi.mutations":1} 2022-03-07 09:53:21,552 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646646201551","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:59502","multi.service_calls":0,"queuetimems":829,"class":"HRegionServer","multi.mutations":2} 2022-03-07 09:53:37,642 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646646217642","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:34920","multi.service_calls":0,"queuetimems":16940,"class":"HRegionServer","multi.mutations":5} 2022-03-07 09:54:16,879 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1585ms No GCs detected 2022-03-07 09:54:29,866 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-07 09:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=300852, hits=292742, hitRatio=97.30%, , cachingAccesses=292872, cachingHits=292433, cachingHitsRatio=99.85%, evictions=19518, evicted=0, evictedPerRun=0.0 2022-03-07 09:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 09:57:18,312 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646646138311","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 5 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:54174","multi.service_calls":0,"queuetimems":84,"class":"HRegionServer","multi.mutations":5} 2022-03-07 09:57:22,445 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646646742445","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:34978","multi.service_calls":0,"queuetimems":4215,"class":"HRegionServer","multi.mutations":1} 2022-03-07 09:57:46,366 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646646466365","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 4 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:54210","multi.service_calls":0,"queuetimems":23922,"class":"HRegionServer","multi.mutations":4} 2022-03-07 09:58:18,509 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646646498509","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:34962","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 09:58:21,553 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646646801552","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:59616","multi.service_calls":0,"queuetimems":59107,"class":"HRegionServer","multi.mutations":1} 2022-03-07 09:58:41,068 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646646521067","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 6 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:34962","multi.service_calls":0,"queuetimems":22558,"class":"HRegionServer","multi.mutations":6} 2022-03-07 10:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=301655, hits=293545, hitRatio=97.31%, , cachingAccesses=293675, cachingHits=293236, cachingHitsRatio=99.85%, evictions=19548, evicted=0, evictedPerRun=0.0 2022-03-07 10:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 10:02:18,231 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646646738230","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 6 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:59616","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":6} 2022-03-07 10:02:18,314 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646647038314","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:34998","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 10:02:39,621 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646646759620","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:54236","multi.service_calls":0,"queuetimems":21389,"class":"HRegionServer","multi.mutations":3} 2022-03-07 10:03:18,510 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646647098510","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:34998","multi.service_calls":0,"queuetimems":32143,"class":"HRegionServer","multi.mutations":1} 2022-03-07 10:03:20,707 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646646500703","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900004,"client":"172.30.41.118:59566","multi.service_calls":0,"queuetimems":2194,"class":"HRegionServer","multi.mutations":4} 2022-03-07 10:03:21,241 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646646801240","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:34982","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 10:04:10,707 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 5052ms No GCs detected 2022-03-07 10:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=302490, hits=294380, hitRatio=97.32%, , cachingAccesses=294510, cachingHits=294071, cachingHitsRatio=99.85%, evictions=19578, evicted=0, evictedPerRun=0.0 2022-03-07 10:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 10:07:18,316 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646647338315","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:54292","multi.service_calls":0,"queuetimems":83,"class":"HRegionServer","multi.mutations":1} 2022-03-07 10:07:22,447 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646647042445","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.118:59690","multi.service_calls":0,"queuetimems":4131,"class":"HRegionServer","multi.mutations":2} 2022-03-07 10:07:46,368 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646647066367","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 6 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:54258","multi.service_calls":0,"queuetimems":28053,"class":"HRegionServer","multi.mutations":6} 2022-03-07 10:07:46,734 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646646766732","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 6 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:34978","multi.service_calls":0,"queuetimems":28502,"class":"HRegionServer","multi.mutations":6} 2022-03-07 10:07:51,023 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1225ms No GCs detected 2022-03-07 10:08:05,103 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2765ms No GCs detected 2022-03-07 10:08:05,103 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-07 10:08:05,163 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646646782531","responsesize":"19835","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 12 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":902632,"client":"172.30.41.118:59616","multi.service_calls":0,"queuetimems":40086,"class":"HRegionServer","multi.mutations":12} 2022-03-07 10:08:05,485 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 336ms. 2022-03-07 10:08:06,079 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: tNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 2 failover attempts. Trying to failover after sleeping for 1457ms. 2022-03-07 10:08:18,512 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646647398512","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:54298","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 10:08:20,709 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646647400708","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.31.0.219:54300","multi.service_calls":0,"queuetimems":2166,"class":"HRegionServer","multi.mutations":2} 2022-03-07 10:08:21,241 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646647401241","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:54298","multi.service_calls":0,"queuetimems":2729,"class":"HRegionServer","multi.mutations":1} 2022-03-07 10:08:37,644 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646646817643","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 7 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:34982","multi.service_calls":0,"queuetimems":16403,"class":"HRegionServer","multi.mutations":7} 2022-03-07 10:08:41,069 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646647121068","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:54258","multi.service_calls":0,"queuetimems":82754,"class":"HRegionServer","multi.mutations":3} 2022-03-07 10:08:50,035 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646646529250","responsesize":"23142","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 14 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200785,"client":"172.31.0.219:54214","multi.service_calls":0,"queuetimems":30741,"class":"HRegionServer","multi.mutations":14} 2022-03-07 10:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=303161, hits=295051, hitRatio=97.32%, , cachingAccesses=295181, cachingHits=294742, cachingHitsRatio=99.85%, evictions=19607, evicted=0, evictedPerRun=0.0 2022-03-07 10:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 10:12:18,233 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646647338232","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:59750","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-07 10:12:22,448 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646647642447","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:54332","multi.service_calls":0,"queuetimems":4130,"class":"HRegionServer","multi.mutations":1} 2022-03-07 10:12:39,622 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646647359621","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 9 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:59750","multi.service_calls":0,"queuetimems":21389,"class":"HRegionServer","multi.mutations":9} 2022-03-07 10:12:57,146 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646646777144","responsesize":"19838","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 12 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200001,"client":"172.31.0.219:54236","multi.service_calls":0,"queuetimems":38913,"class":"HRegionServer","multi.mutations":12} 2022-03-07 10:13:23,974 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646647701242","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300451,"client":"172.30.41.155:35036","multi.service_calls":0,"queuetimems":11299,"class":"HRegionServer","multi.mutations":1} 2022-03-07 10:13:23,981 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646646500722","responsesize":"24797","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 15 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1503259,"client":"172.30.41.118:59566","multi.service_calls":0,"queuetimems":2213,"class":"HRegionServer","multi.mutations":15} 2022-03-07 10:13:23,982 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1790ms No GCs detected 2022-03-07 10:13:31,953 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1459ms No GCs detected 2022-03-07 10:13:37,645 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646647717644","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:59818","multi.service_calls":0,"queuetimems":24111,"class":"HRegionServer","multi.mutations":1} 2022-03-07 10:13:40,211 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-07 10:13:41,070 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646647721069","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:54332","multi.service_calls":0,"queuetimems":54700,"class":"HRegionServer","multi.mutations":2} 2022-03-07 10:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=303918, hits=295808, hitRatio=97.33%, , cachingAccesses=295938, cachingHits=295499, cachingHitsRatio=99.85%, evictions=19637, evicted=0, evictedPerRun=0.0 2022-03-07 10:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 10:17:18,235 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646647938234","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:59872","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 10:17:18,318 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646647638317","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:35036","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 10:17:46,735 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646647666734","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 6 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:35036","multi.service_calls":0,"queuetimems":28417,"class":"HRegionServer","multi.mutations":6} 2022-03-07 10:17:55,369 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1830ms No GCs detected 2022-03-07 10:17:57,146 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646647977146","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:54346","multi.service_calls":0,"queuetimems":38912,"class":"HRegionServer","multi.mutations":1} 2022-03-07 10:18:05,296 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2817ms No GCs detected 2022-03-07 10:18:18,513 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646647698512","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:35036","multi.service_calls":0,"queuetimems":32143,"class":"HRegionServer","multi.mutations":2} 2022-03-07 10:18:23,993 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646648003993","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:35054","multi.service_calls":0,"queuetimems":6,"class":"HRegionServer","multi.mutations":1} 2022-03-07 10:18:37,645 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646648017645","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:35054","multi.service_calls":0,"queuetimems":13658,"class":"HRegionServer","multi.mutations":1} 2022-03-07 10:18:41,071 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646648021070","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:54348","multi.service_calls":0,"queuetimems":17094,"class":"HRegionServer","multi.mutations":3} 2022-03-07 10:18:50,036 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646647730035","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 8 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:54332","multi.service_calls":0,"queuetimems":63666,"class":"HRegionServer","multi.mutations":8} 2022-03-07 10:20:17,186 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2953ms No GCs detected 2022-03-07 10:21:44,438 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=304536, hits=296426, hitRatio=97.34%, , cachingAccesses=296556, cachingHits=296117, cachingHitsRatio=99.85%, evictions=19667, evicted=0, evictedPerRun=0.0 2022-03-07 10:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 10:22:18,236 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646648238236","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:59956","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 10:22:18,319 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646648238318","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:54362","multi.service_calls":0,"queuetimems":82,"class":"HRegionServer","multi.mutations":1} 2022-03-07 10:22:39,623 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646647959623","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:59872","multi.service_calls":0,"queuetimems":17174,"class":"HRegionServer","multi.mutations":4} 2022-03-07 10:22:46,735 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646648266735","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:59956","multi.service_calls":0,"queuetimems":28416,"class":"HRegionServer","multi.mutations":1} 2022-03-07 10:22:57,148 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646648277147","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:35070","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 10:23:05,297 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646647685163","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 5 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900134,"client":"172.31.0.219:54332","multi.service_calls":0,"queuetimems":46845,"class":"HRegionServer","multi.mutations":5} 2022-03-07 10:23:18,514 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646648298514","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:59958","multi.service_calls":0,"queuetimems":21367,"class":"HRegionServer","multi.mutations":2} 2022-03-07 10:23:23,972 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646647700709","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 6 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":903262,"client":"172.30.41.118:59818","multi.service_calls":0,"queuetimems":62392,"class":"HRegionServer","multi.mutations":6} 2022-03-07 10:23:23,972 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646647101553","responsesize":"26449","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 16 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1502419,"client":"172.30.41.118:59690","multi.service_calls":0,"queuetimems":63239,"class":"HRegionServer","multi.mutations":16} 2022-03-07 10:23:23,985 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646648003984","responsesize":"11572","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:54348","multi.service_calls":0,"queuetimems":8,"class":"HRegionServer","multi.mutations":7} 2022-03-07 10:23:23,993 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646648303993","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:35070","multi.service_calls":0,"queuetimems":26846,"class":"HRegionServer","multi.mutations":1} 2022-03-07 10:23:37,646 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646648317645","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:35070","multi.service_calls":0,"queuetimems":40498,"class":"HRegionServer","multi.mutations":1} 2022-03-07 10:23:41,072 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646648321072","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:35070","multi.service_calls":0,"queuetimems":43925,"class":"HRegionServer","multi.mutations":1} 2022-03-07 10:23:50,036 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646648330036","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:35070","multi.service_calls":0,"queuetimems":52888,"class":"HRegionServer","multi.mutations":1} 2022-03-07 10:24:15,971 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2235ms No GCs detected 2022-03-07 10:25:21,775 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-07 10:25:22,798 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 300ms. 2022-03-07 10:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=305312, hits=297202, hitRatio=97.34%, , cachingAccesses=297332, cachingHits=296893, cachingHitsRatio=99.85%, evictions=19697, evicted=0, evictedPerRun=0.0 2022-03-07 10:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 10:27:18,238 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646648538237","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:60014","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 10:27:22,450 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646647942449","responsesize":"18183","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 11 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.30.41.118:59872","multi.service_calls":0,"queuetimems":4215,"class":"HRegionServer","multi.mutations":11} 2022-03-07 10:27:39,625 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646648559623","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.155:35082","multi.service_calls":0,"queuetimems":21385,"class":"HRegionServer","multi.mutations":1} 2022-03-07 10:27:46,578 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646647666368","responsesize":"21490","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 13 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200210,"client":"172.30.41.118:59818","multi.service_calls":0,"queuetimems":28051,"class":"HRegionServer","multi.mutations":13} 2022-03-07 10:27:53,885 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-07 10:28:18,515 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646648598515","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:54378","multi.service_calls":0,"queuetimems":21366,"class":"HRegionServer","multi.mutations":1} 2022-03-07 10:28:23,973 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646648603972","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:60026","multi.service_calls":0,"queuetimems":18674,"class":"HRegionServer","multi.mutations":2} 2022-03-07 10:28:23,994 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646648603993","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.30.41.118:60024","multi.service_calls":0,"queuetimems":26814,"class":"HRegionServer","multi.mutations":2} 2022-03-07 10:28:25,247 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-07 10:28:37,646 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646648617646","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:60026","multi.service_calls":0,"queuetimems":32348,"class":"HRegionServer","multi.mutations":1} 2022-03-07 10:28:41,072 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646648621072","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:35084","multi.service_calls":0,"queuetimems":35774,"class":"HRegionServer","multi.mutations":1} 2022-03-07 10:28:51,342 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646648631342","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:54380","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 10:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=306002, hits=297892, hitRatio=97.35%, , cachingAccesses=298022, cachingHits=297583, cachingHitsRatio=99.85%, evictions=19727, evicted=0, evictedPerRun=0.0 2022-03-07 10:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-07 10:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 10:32:18,240 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646648838239","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:35098","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 10:32:22,451 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646648842451","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:35098","multi.service_calls":0,"queuetimems":4212,"class":"HRegionServer","multi.mutations":2} 2022-03-07 10:32:47,193 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646648566736","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 9 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600457,"client":"172.31.0.219:54376","multi.service_calls":0,"queuetimems":28498,"class":"HRegionServer","multi.mutations":9} 2022-03-07 10:33:18,516 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646648898516","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:60104","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 10:33:23,994 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646648903994","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:54396","multi.service_calls":0,"queuetimems":5478,"class":"HRegionServer","multi.mutations":1} 2022-03-07 10:33:25,774 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-07 10:33:37,647 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646648917647","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:54396","multi.service_calls":0,"queuetimems":19131,"class":"HRegionServer","multi.mutations":1} 2022-03-07 10:33:51,343 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646648931342","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:54396","multi.service_calls":0,"queuetimems":32826,"class":"HRegionServer","multi.mutations":1} 2022-03-07 10:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=306757, hits=298647, hitRatio=97.36%, , cachingAccesses=298777, cachingHits=298338, cachingHitsRatio=99.85%, evictions=19757, evicted=0, evictedPerRun=0.0 2022-03-07 10:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 10:37:18,321 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646648538319","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 7 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.118:60014","multi.service_calls":0,"queuetimems":81,"class":"HRegionServer","multi.mutations":7} 2022-03-07 10:37:37,686 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1344ms No GCs detected 2022-03-07 10:37:39,626 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646648859625","responsesize":"11572","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:60094","multi.service_calls":0,"queuetimems":21386,"class":"HRegionServer","multi.mutations":7} 2022-03-07 10:37:47,193 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646649167193","responsesize":"6613","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:35116","multi.service_calls":0,"queuetimems":24741,"class":"HRegionServer","multi.mutations":4} 2022-03-07 10:37:57,150 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646648577149","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:60026","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-07 10:38:19,352 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 5157ms No GCs detected 2022-03-07 10:38:23,975 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646648903973","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 4 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:35100","multi.service_calls":0,"queuetimems":5457,"class":"HRegionServer","multi.mutations":4} 2022-03-07 10:38:51,344 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646649231344","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:60188","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 10:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=307505, hits=299395, hitRatio=97.36%, , cachingAccesses=299525, cachingHits=299086, cachingHitsRatio=99.85%, evictions=19787, evicted=0, evictedPerRun=0.0 2022-03-07 10:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 10:42:18,322 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646649438321","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:60244","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 10:42:22,454 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646649142452","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:35116","multi.service_calls":0,"queuetimems":4211,"class":"HRegionServer","multi.mutations":2} 2022-03-07 10:42:47,194 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646649467193","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:35134","multi.service_calls":0,"queuetimems":28872,"class":"HRegionServer","multi.mutations":1} 2022-03-07 10:43:05,299 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646648585297","responsesize":"13230","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 8 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200002,"client":"172.30.41.155:35084","multi.service_calls":0,"queuetimems":8148,"class":"HRegionServer","multi.mutations":8} 2022-03-07 10:43:19,373 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646649198517","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600856,"client":"172.30.41.118:60178","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-07 10:43:23,973 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646648603972","responsesize":"19838","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 12 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200001,"client":"172.30.41.118:60026","multi.service_calls":0,"queuetimems":26823,"class":"HRegionServer","multi.mutations":12} 2022-03-07 10:43:23,986 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646648603985","responsesize":"26446","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 16 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200001,"client":"172.31.0.219:54378","multi.service_calls":0,"queuetimems":26836,"class":"HRegionServer","multi.mutations":16} 2022-03-07 10:43:37,648 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646649217647","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:35118","multi.service_calls":0,"queuetimems":19130,"class":"HRegionServer","multi.mutations":3} 2022-03-07 10:43:51,345 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646649531344","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:35136","multi.service_calls":0,"queuetimems":27368,"class":"HRegionServer","multi.mutations":1} 2022-03-07 10:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=308117, hits=300007, hitRatio=97.37%, , cachingAccesses=300137, cachingHits=299698, cachingHitsRatio=99.85%, evictions=19817, evicted=0, evictedPerRun=0.0 2022-03-07 10:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 10:47:18,242 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646649138240","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 6 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.118:60166","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":6} 2022-03-07 10:47:22,454 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646649742454","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:35152","multi.service_calls":0,"queuetimems":4131,"class":"HRegionServer","multi.mutations":1} 2022-03-07 10:47:39,631 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646649459627","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600004,"client":"172.30.41.118:60244","multi.service_calls":0,"queuetimems":21305,"class":"HRegionServer","multi.mutations":4} 2022-03-07 10:47:46,580 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646648866578","responsesize":"14882","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 9 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200002,"client":"172.30.41.118:60094","multi.service_calls":0,"queuetimems":28339,"class":"HRegionServer","multi.mutations":9} 2022-03-07 10:47:57,151 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646649477150","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:35134","multi.service_calls":0,"queuetimems":38828,"class":"HRegionServer","multi.mutations":2} 2022-03-07 10:48:05,300 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646649785299","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.30.41.118:60318","multi.service_calls":0,"queuetimems":46946,"class":"HRegionServer","multi.mutations":2} 2022-03-07 10:48:19,375 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646649799374","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:60320","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 10:48:23,976 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646649803974","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.155:35154","multi.service_calls":0,"queuetimems":4600,"class":"HRegionServer","multi.mutations":1} 2022-03-07 10:48:23,976 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646649503975","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:60258","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 10:48:23,987 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646649803986","responsesize":"6613","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:60320","multi.service_calls":0,"queuetimems":4612,"class":"HRegionServer","multi.mutations":4} 2022-03-07 10:48:23,995 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646649203994","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 8 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:60178","multi.service_calls":0,"queuetimems":5477,"class":"HRegionServer","multi.mutations":8} 2022-03-07 10:48:41,074 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646648921073","responsesize":"23142","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 14 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200001,"client":"172.30.41.118:60104","multi.service_calls":0,"queuetimems":22557,"class":"HRegionServer","multi.mutations":14} 2022-03-07 10:48:51,345 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646649831345","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:60320","multi.service_calls":0,"queuetimems":31971,"class":"HRegionServer","multi.mutations":1} 2022-03-07 10:50:08,493 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3127ms No GCs detected 2022-03-07 10:50:26,584 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1278ms No GCs detected 2022-03-07 10:50:33,535 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2950ms No GCs detected 2022-03-07 10:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=308823, hits=300713, hitRatio=97.37%, , cachingAccesses=300843, cachingHits=300404, cachingHitsRatio=99.85%, evictions=19847, evicted=0, evictedPerRun=0.0 2022-03-07 10:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 10:52:18,324 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646649738323","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:35152","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 10:52:57,152 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646650077151","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:54462","multi.service_calls":0,"queuetimems":38908,"class":"HRegionServer","multi.mutations":1} 2022-03-07 10:53:03,307 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646650083307","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:60390","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 10:53:09,780 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646650089780","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300000,"client":"172.30.41.118:60392","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 10:53:33,347 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646650113346","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.30.41.118:60394","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 10:53:39,816 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646650119815","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300000,"client":"172.30.41.118:60396","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 10:53:39,909 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646650119909","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:60398","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-07 10:53:41,075 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646650121075","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:60398","multi.service_calls":0,"queuetimems":1166,"class":"HRegionServer","multi.mutations":1} 2022-03-07 10:53:51,346 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646650131346","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:35170","multi.service_calls":0,"queuetimems":10271,"class":"HRegionServer","multi.mutations":1} 2022-03-07 10:56:16,415 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1284ms No GCs detected 2022-03-07 10:56:28,177 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3685ms No GCs detected 2022-03-07 10:56:34,499 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-07 10:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=309532, hits=301422, hitRatio=97.38%, , cachingAccesses=301552, cachingHits=301113, cachingHitsRatio=99.85%, evictions=19877, evicted=0, evictedPerRun=0.0 2022-03-07 10:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 10:57:03,644 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1864ms No GCs detected 2022-03-07 10:57:18,814 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646650038243","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600571,"client":"172.30.41.118:60382","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-07 10:57:19,453 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4781ms No GCs detected 2022-03-07 10:57:24,135 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2659ms No GCs detected 2022-03-07 10:57:24,144 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646650042455","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":601680,"client":"172.30.41.118:60382","multi.service_calls":0,"queuetimems":4212,"class":"HRegionServer","multi.mutations":3} 2022-03-07 10:57:39,632 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646650059631","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:35168","multi.service_calls":0,"queuetimems":17176,"class":"HRegionServer","multi.mutations":2} 2022-03-07 10:57:47,195 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646649767194","responsesize":"19835","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 12 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.31.0.219:54446","multi.service_calls":0,"queuetimems":28871,"class":"HRegionServer","multi.mutations":12} 2022-03-07 10:57:57,153 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646650377152","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:60458","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 10:58:09,781 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646650389780","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:60458","multi.service_calls":0,"queuetimems":12628,"class":"HRegionServer","multi.mutations":1} 2022-03-07 10:58:23,977 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646650103976","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:60382","multi.service_calls":0,"queuetimems":61521,"class":"HRegionServer","multi.mutations":2} 2022-03-07 10:58:33,347 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646650413347","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:60458","multi.service_calls":0,"queuetimems":30039,"class":"HRegionServer","multi.mutations":1} 2022-03-07 10:58:39,816 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646650419816","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:60458","multi.service_calls":0,"queuetimems":36508,"class":"HRegionServer","multi.mutations":2} 2022-03-07 10:58:39,910 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646650419909","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:35186","multi.service_calls":0,"queuetimems":36601,"class":"HRegionServer","multi.mutations":2} 2022-03-07 10:58:41,076 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646650421075","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:60458","multi.service_calls":0,"queuetimems":43922,"class":"HRegionServer","multi.mutations":1} 2022-03-07 10:58:51,347 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646650431346","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:35186","multi.service_calls":0,"queuetimems":54193,"class":"HRegionServer","multi.mutations":2} 2022-03-07 11:01:35,690 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3846ms No GCs detected 2022-03-07 11:01:38,221 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-07 11:01:38,417 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 499ms. 2022-03-07 11:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=310277, hits=302167, hitRatio=97.39%, , cachingAccesses=302297, cachingHits=301858, cachingHitsRatio=99.85%, evictions=19907, evicted=0, evictedPerRun=0.0 2022-03-07 11:01:47,384 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2662ms No GCs detected 2022-03-07 11:01:49,183 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 11:02:18,816 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646650338325","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600491,"client":"172.30.41.118:60446","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 11:02:24,145 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646650644144","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:60500","multi.service_calls":0,"queuetimems":4689,"class":"HRegionServer","multi.mutations":1} 2022-03-07 11:02:47,196 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646650667196","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:60500","multi.service_calls":0,"queuetimems":27740,"class":"HRegionServer","multi.mutations":1} 2022-03-07 11:02:57,154 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646650677153","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:54526","multi.service_calls":0,"queuetimems":37694,"class":"HRegionServer","multi.mutations":1} 2022-03-07 11:03:03,309 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646650383308","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:35186","multi.service_calls":0,"queuetimems":6156,"class":"HRegionServer","multi.mutations":3} 2022-03-07 11:03:09,782 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646650689781","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:54526","multi.service_calls":0,"queuetimems":50322,"class":"HRegionServer","multi.mutations":1} 2022-03-07 11:03:23,979 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646650703978","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:60508","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 11:03:23,979 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646650103976","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 6 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900003,"client":"172.31.0.219:54462","multi.service_calls":0,"queuetimems":65733,"class":"HRegionServer","multi.mutations":6} 2022-03-07 11:03:37,650 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646649817648","responsesize":"16534","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 10 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.155:35154","multi.service_calls":0,"queuetimems":18274,"class":"HRegionServer","multi.mutations":10} 2022-03-07 11:03:39,817 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646650719816","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:54536","multi.service_calls":0,"queuetimems":15838,"class":"HRegionServer","multi.mutations":1} 2022-03-07 11:03:39,910 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646650719910","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:54536","multi.service_calls":0,"queuetimems":15931,"class":"HRegionServer","multi.mutations":1} 2022-03-07 11:03:51,348 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646650731347","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:35208","multi.service_calls":0,"queuetimems":27369,"class":"HRegionServer","multi.mutations":2} 2022-03-07 11:04:25,450 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2050ms No GCs detected 2022-03-07 11:04:29,925 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3974ms No GCs detected 2022-03-07 11:05:59,162 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3946ms No GCs detected 2022-03-07 11:06:10,131 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-07 11:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=311004, hits=302894, hitRatio=97.39%, , cachingAccesses=303024, cachingHits=302585, cachingHitsRatio=99.86%, evictions=19937, evicted=0, evictedPerRun=0.0 2022-03-07 11:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 11:06:56,126 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2991ms No GCs detected 2022-03-07 11:07:19,466 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646650639465","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 5 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:35206","multi.service_calls":0,"queuetimems":647,"class":"HRegionServer","multi.mutations":5} 2022-03-07 11:07:48,830 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646650968829","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:35234","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 11:07:57,154 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646650977154","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:35234","multi.service_calls":0,"queuetimems":8325,"class":"HRegionServer","multi.mutations":2} 2022-03-07 11:08:24,010 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646651004010","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.155:35238","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 11:08:33,349 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646650713348","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:35208","multi.service_calls":0,"queuetimems":9370,"class":"HRegionServer","multi.mutations":2} 2022-03-07 11:08:33,744 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646651013744","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:60562","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 11:08:37,651 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646651017650","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:54562","multi.service_calls":0,"queuetimems":34340,"class":"HRegionServer","multi.mutations":1} 2022-03-07 11:08:39,912 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646651019911","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:60562","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 11:10:10,739 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1205ms No GCs detected 2022-03-07 11:10:13,045 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1223ms No GCs detected 2022-03-07 11:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=311767, hits=303657, hitRatio=97.40%, , cachingAccesses=303787, cachingHits=303348, cachingHitsRatio=99.86%, evictions=19967, evicted=0, evictedPerRun=0.0 2022-03-07 11:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 11:12:06,900 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1751ms No GCs detected 2022-03-07 11:12:08,508 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1107ms No GCs detected 2022-03-07 11:12:12,005 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-07 11:12:19,468 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646651239467","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:35264","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 11:12:25,131 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1001ms No GCs detected 2022-03-07 11:12:26,645 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1014ms No GCs detected 2022-03-07 11:12:33,729 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1085ms No GCs detected 2022-03-07 11:12:39,661 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646650659632","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 6 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900029,"client":"172.30.41.155:35206","multi.service_calls":0,"queuetimems":20178,"class":"HRegionServer","multi.mutations":6} 2022-03-07 11:12:49,665 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-07 11:12:57,938 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4276ms No GCs detected 2022-03-07 11:13:03,310 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646650983309","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:54562","multi.service_calls":0,"queuetimems":44491,"class":"HRegionServer","multi.mutations":3} 2022-03-07 11:13:09,783 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646650989782","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:54562","multi.service_calls":0,"queuetimems":50964,"class":"HRegionServer","multi.mutations":3} 2022-03-07 11:13:20,014 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-07 11:13:33,350 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646651313350","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:60610","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 11:13:37,651 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646651317651","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300000,"client":"172.30.41.118:60610","multi.service_calls":0,"queuetimems":4301,"class":"HRegionServer","multi.mutations":3} 2022-03-07 11:13:39,818 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646651019817","responsesize":"16528","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 10 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":600001,"client":"172.31.0.219:54562","multi.service_calls":0,"queuetimems":36507,"class":"HRegionServer","multi.mutations":10} 2022-03-07 11:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=312652, hits=304542, hitRatio=97.41%, , cachingAccesses=304672, cachingHits=304233, cachingHitsRatio=99.86%, evictions=19997, evicted=0, evictedPerRun=0.0 2022-03-07 11:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 11:17:18,822 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646650938817","responsesize":"14879","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 9 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900005,"client":"172.30.41.118:60548","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":9} 2022-03-07 11:17:25,131 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646650944145","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 4 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900986,"client":"172.30.41.155:35228","multi.service_calls":0,"queuetimems":5327,"class":"HRegionServer","multi.mutations":4} 2022-03-07 11:17:39,662 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646651559662","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:60650","multi.service_calls":0,"queuetimems":20193,"class":"HRegionServer","multi.mutations":1} 2022-03-07 11:17:48,832 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646651268831","responsesize":"11572","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 7 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:35264","multi.service_calls":0,"queuetimems":29364,"class":"HRegionServer","multi.mutations":7} 2022-03-07 11:18:09,783 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646651589783","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:54614","multi.service_calls":0,"queuetimems":6471,"class":"HRegionServer","multi.mutations":1} 2022-03-07 11:18:33,351 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646651613350","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:35310","multi.service_calls":0,"queuetimems":29843,"class":"HRegionServer","multi.mutations":1} 2022-03-07 11:18:33,746 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646651313745","responsesize":"11572","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:35274","multi.service_calls":0,"queuetimems":395,"class":"HRegionServer","multi.mutations":7} 2022-03-07 11:18:37,652 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646651617652","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300000,"client":"172.30.41.118:60656","multi.service_calls":0,"queuetimems":4273,"class":"HRegionServer","multi.mutations":1} 2022-03-07 11:18:41,078 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646650721076","responsesize":"24794","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 15 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.118:60508","multi.service_calls":0,"queuetimems":17098,"class":"HRegionServer","multi.mutations":15} 2022-03-07 11:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=313514, hits=305404, hitRatio=97.41%, , cachingAccesses=305534, cachingHits=305095, cachingHitsRatio=99.86%, evictions=20027, evicted=0, evictedPerRun=0.0 2022-03-07 11:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 11:22:19,470 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646651539469","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:35306","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 11:22:25,132 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646651845131","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:35346","multi.service_calls":0,"queuetimems":6308,"class":"HRegionServer","multi.mutations":1} 2022-03-07 11:22:39,663 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646651859663","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:35346","multi.service_calls":0,"queuetimems":20840,"class":"HRegionServer","multi.mutations":2} 2022-03-07 11:23:03,313 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646651583312","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 6 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:60654","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":6} 2022-03-07 11:23:24,013 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646651304011","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 4 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.31.0.219:54588","multi.service_calls":0,"queuetimems":64543,"class":"HRegionServer","multi.mutations":4} 2022-03-07 11:23:33,352 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646651913352","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:35352","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 11:23:33,747 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646651913747","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:60702","multi.service_calls":0,"queuetimems":395,"class":"HRegionServer","multi.mutations":2} 2022-03-07 11:23:37,653 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646651917652","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:54652","multi.service_calls":0,"queuetimems":4300,"class":"HRegionServer","multi.mutations":2} 2022-03-07 11:23:39,820 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646651619819","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:35316","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 11:23:51,350 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646651031348","responsesize":"23142","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 14 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.118:60562","multi.service_calls":0,"queuetimems":17604,"class":"HRegionServer","multi.mutations":14} 2022-03-07 11:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=314396, hits=306286, hitRatio=97.42%, , cachingAccesses=306416, cachingHits=305977, cachingHitsRatio=99.86%, evictions=20057, evicted=0, evictedPerRun=0.0 2022-03-07 11:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 11:27:18,824 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646651838823","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:54644","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 11:27:19,471 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646652139471","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:60744","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 11:27:25,134 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646652145132","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.155:35380","multi.service_calls":0,"queuetimems":5661,"class":"HRegionServer","multi.mutations":1} 2022-03-07 11:27:48,833 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646651868832","responsesize":"11572","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:35346","multi.service_calls":0,"queuetimems":30009,"class":"HRegionServer","multi.mutations":7} 2022-03-07 11:27:57,915 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646651277155","responsesize":"24794","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 15 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200760,"client":"172.30.41.118:60598","multi.service_calls":0,"queuetimems":37688,"class":"HRegionServer","multi.mutations":15} 2022-03-07 11:28:03,314 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646652183313","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:54678","multi.service_calls":0,"queuetimems":38180,"class":"HRegionServer","multi.mutations":3} 2022-03-07 11:28:09,784 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646651889783","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:54644","multi.service_calls":0,"queuetimems":50960,"class":"HRegionServer","multi.mutations":2} 2022-03-07 11:28:33,748 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646652213748","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:54682","multi.service_calls":0,"queuetimems":395,"class":"HRegionServer","multi.mutations":1} 2022-03-07 11:28:39,820 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646652219820","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:54682","multi.service_calls":0,"queuetimems":15806,"class":"HRegionServer","multi.mutations":1} 2022-03-07 11:28:39,913 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646651319912","responsesize":"23142","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 14 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200001,"client":"172.30.41.118:60610","multi.service_calls":0,"queuetimems":6167,"class":"HRegionServer","multi.mutations":14} 2022-03-07 11:28:41,079 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646651921078","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 5 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:60702","multi.service_calls":0,"queuetimems":7726,"class":"HRegionServer","multi.mutations":5} 2022-03-07 11:29:06,267 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1356ms No GCs detected 2022-03-07 11:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=315281, hits=307171, hitRatio=97.43%, , cachingAccesses=307301, cachingHits=306862, cachingHitsRatio=99.86%, evictions=20087, evicted=0, evictedPerRun=0.0 2022-03-07 11:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-07 11:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 11:32:08,668 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1590ms No GCs detected 2022-03-07 11:32:18,826 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646652438825","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:60814","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 11:32:19,557 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646652439472","responsesize":"8265","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 5 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300084,"client":"172.30.41.118:60814","multi.service_calls":0,"queuetimems":647,"class":"HRegionServer","multi.mutations":5} 2022-03-07 11:32:22,644 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-07 11:32:39,665 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646652159663","responsesize":"18180","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 11 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":600001,"client":"172.31.0.219:54678","multi.service_calls":0,"queuetimems":20192,"class":"HRegionServer","multi.mutations":11} 2022-03-07 11:32:48,834 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646652468833","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:54696","multi.service_calls":0,"queuetimems":30007,"class":"HRegionServer","multi.mutations":2} 2022-03-07 11:32:57,916 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646652477915","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:60818","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 11:33:09,785 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646652489784","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.30.41.118:60820","multi.service_calls":0,"queuetimems":11838,"class":"HRegionServer","multi.mutations":2} 2022-03-07 11:33:24,016 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646652204014","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 4 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.118:60752","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-07 11:33:37,654 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646652217653","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 5 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:35386","multi.service_calls":0,"queuetimems":13639,"class":"HRegionServer","multi.mutations":5} 2022-03-07 11:33:39,935 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646652519934","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300001,"client":"172.30.41.118:60828","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 11:33:41,079 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646652521079","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:54698","multi.service_calls":0,"queuetimems":37764,"class":"HRegionServer","multi.mutations":1} 2022-03-07 11:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=316035, hits=307925, hitRatio=97.43%, , cachingAccesses=308055, cachingHits=307616, cachingHitsRatio=99.86%, evictions=20117, evicted=0, evictedPerRun=0.0 2022-03-07 11:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 11:37:19,637 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646652739636","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:35456","multi.service_calls":0,"queuetimems":809,"class":"HRegionServer","multi.mutations":1} 2022-03-07 11:37:25,355 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646652445134","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600221,"client":"172.30.41.155:35420","multi.service_calls":0,"queuetimems":6308,"class":"HRegionServer","multi.mutations":4} 2022-03-07 11:37:39,665 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646652759665","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:60882","multi.service_calls":0,"queuetimems":19694,"class":"HRegionServer","multi.mutations":1} 2022-03-07 11:37:57,916 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646652777916","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:54716","multi.service_calls":0,"queuetimems":38279,"class":"HRegionServer","multi.mutations":1} 2022-03-07 11:38:09,786 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646652789785","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:60882","multi.service_calls":0,"queuetimems":49668,"class":"HRegionServer","multi.mutations":1} 2022-03-07 11:38:24,016 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646652804016","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:54716","multi.service_calls":0,"queuetimems":64049,"class":"HRegionServer","multi.mutations":1} 2022-03-07 11:38:33,749 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646652513748","responsesize":"11572","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 7 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:54698","multi.service_calls":0,"queuetimems":35832,"class":"HRegionServer","multi.mutations":7} 2022-03-07 11:38:37,655 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646652817655","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:35458","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 11:38:39,822 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646652519821","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 6 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:60826","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":6} 2022-03-07 11:38:39,935 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646652819935","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:35458","multi.service_calls":0,"queuetimems":2280,"class":"HRegionServer","multi.mutations":2} 2022-03-07 11:40:16,648 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1079ms No GCs detected 2022-03-07 11:40:51,872 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 5471ms No GCs detected 2022-03-07 11:40:54,970 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2597ms No GCs detected 2022-03-07 11:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=316810, hits=308700, hitRatio=97.44%, , cachingAccesses=308830, cachingHits=308391, cachingHitsRatio=99.86%, evictions=20147, evicted=0, evictedPerRun=0.0 2022-03-07 11:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 11:42:18,828 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646652738827","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:60882","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 11:42:48,835 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646652768834","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:54716","multi.service_calls":0,"queuetimems":30007,"class":"HRegionServer","multi.mutations":2} 2022-03-07 11:42:57,918 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646653077917","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:35474","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 11:43:09,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646653089786","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:60956","multi.service_calls":0,"queuetimems":11869,"class":"HRegionServer","multi.mutations":3} 2022-03-07 11:43:33,354 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646652213352","responsesize":"24794","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 15 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Dev ","processingtimems":1200002,"client":"172.30.41.118:60752","multi.service_calls":0,"queuetimems":9338,"class":"HRegionServer","multi.mutations":15} 2022-03-07 11:43:41,081 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646652821080","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:54718","multi.service_calls":0,"queuetimems":3425,"class":"HRegionServer","multi.mutations":3} 2022-03-07 11:43:51,352 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646652231350","responsesize":"23142","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 14 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.31.0.219:54682","multi.service_calls":0,"queuetimems":17997,"class":"HRegionServer","multi.mutations":14} 2022-03-07 11:46:02,171 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-07 11:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=317463, hits=309353, hitRatio=97.45%, , cachingAccesses=309483, cachingHits=309044, cachingHitsRatio=99.86%, evictions=20177, evicted=0, evictedPerRun=0.0 2022-03-07 11:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 11:47:19,639 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646653039638","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:60950","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 11:47:25,358 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646653045356","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.155:35472","multi.service_calls":0,"queuetimems":5718,"class":"HRegionServer","multi.mutations":4} 2022-03-07 11:47:58,648 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646653377918","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300730,"client":"172.30.41.118:32786","multi.service_calls":0,"queuetimems":39089,"class":"HRegionServer","multi.mutations":2} 2022-03-07 11:48:03,316 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646652483314","responsesize":"11578","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 7 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200002,"client":"172.31.0.219:54698","multi.service_calls":0,"queuetimems":5398,"class":"HRegionServer","multi.mutations":7} 2022-03-07 11:48:09,788 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646653389787","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:32786","multi.service_calls":0,"queuetimems":25049,"class":"HRegionServer","multi.mutations":1} 2022-03-07 11:48:12,456 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1419ms No GCs detected 2022-03-07 11:48:35,654 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646653413354","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":302300,"client":"172.30.41.118:32786","multi.service_calls":0,"queuetimems":39354,"class":"HRegionServer","multi.mutations":1} 2022-03-07 11:48:35,654 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646653113749","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 6 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":601905,"client":"172.31.0.219:54736","multi.service_calls":0,"queuetimems":35831,"class":"HRegionServer","multi.mutations":6} 2022-03-07 11:48:37,699 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646653117656","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600042,"client":"172.30.41.155:35474","multi.service_calls":0,"queuetimems":27870,"class":"HRegionServer","multi.mutations":3} 2022-03-07 11:48:39,936 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646653119935","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:54736","multi.service_calls":0,"queuetimems":30148,"class":"HRegionServer","multi.mutations":3} 2022-03-07 11:49:05,188 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-07 11:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=318117, hits=310007, hitRatio=97.45%, , cachingAccesses=310137, cachingHits=309698, cachingHitsRatio=99.86%, evictions=20207, evicted=0, evictedPerRun=0.0 2022-03-07 11:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 11:52:18,830 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646653338829","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:54754","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 11:52:19,641 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646653639640","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:32854","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 11:52:25,359 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646653645358","responsesize":"9917","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:32854","multi.service_calls":0,"queuetimems":5718,"class":"HRegionServer","multi.mutations":6} 2022-03-07 11:52:39,667 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646653059665","responsesize":"14879","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 9 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.118:60950","multi.service_calls":0,"queuetimems":20027,"class":"HRegionServer","multi.mutations":9} 2022-03-07 11:53:03,317 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646653683316","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:35504","multi.service_calls":0,"queuetimems":43676,"class":"HRegionServer","multi.mutations":1} 2022-03-07 11:53:24,018 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646653104016","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 7 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.155:35474","multi.service_calls":0,"queuetimems":26098,"class":"HRegionServer","multi.mutations":7} 2022-03-07 11:53:35,656 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646653715656","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:32862","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 11:53:35,657 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646653715656","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:32862","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 11:53:38,114 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646653717699","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300415,"client":"172.30.41.118:32862","multi.service_calls":0,"queuetimems":2043,"class":"HRegionServer","multi.mutations":1} 2022-03-07 11:53:39,501 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2622ms No GCs detected 2022-03-07 11:53:39,951 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646653719936","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300015,"client":"172.30.41.118:32862","multi.service_calls":0,"queuetimems":4280,"class":"HRegionServer","multi.mutations":1} 2022-03-07 11:53:41,151 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646653421082","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600068,"client":"172.30.41.118:32804","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 11:54:24,478 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2126ms No GCs detected 2022-03-07 11:54:27,017 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1039ms No GCs detected 2022-03-07 11:54:36,769 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1310ms No GCs detected 2022-03-07 11:54:38,831 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1562ms No GCs detected 2022-03-07 11:55:09,002 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-07 11:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=318795, hits=310685, hitRatio=97.46%, , cachingAccesses=310815, cachingHits=310376, cachingHitsRatio=99.86%, evictions=20236, evicted=0, evictedPerRun=0.0 2022-03-07 11:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 11:57:39,667 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646653959667","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:35518","multi.service_calls":0,"queuetimems":20026,"class":"HRegionServer","multi.mutations":1} 2022-03-07 11:57:48,836 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646653368835","responsesize":"14879","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 9 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:32786","multi.service_calls":0,"queuetimems":30006,"class":"HRegionServer","multi.mutations":9} 2022-03-07 11:58:03,318 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646653983317","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:32936","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 11:58:24,019 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646654004019","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:35518","multi.service_calls":0,"queuetimems":64378,"class":"HRegionServer","multi.mutations":1} 2022-03-07 11:58:35,657 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646654015656","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.30.41.155:35454","multi.service_calls":0,"queuetimems":25547,"class":"HRegionServer","multi.mutations":2} 2022-03-07 11:58:35,657 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646654015657","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:32936","multi.service_calls":0,"queuetimems":20787,"class":"HRegionServer","multi.mutations":1} 2022-03-07 11:58:38,114 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646654018114","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:35518","multi.service_calls":0,"queuetimems":26870,"class":"HRegionServer","multi.mutations":1} 2022-03-07 11:58:39,824 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646653119822","responsesize":"24794","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 15 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Dev ","processingtimems":1200002,"client":"172.30.41.118:60956","multi.service_calls":0,"queuetimems":30035,"class":"HRegionServer","multi.mutations":15} 2022-03-07 11:58:39,952 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646654019951","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:35518","multi.service_calls":0,"queuetimems":17129,"class":"HRegionServer","multi.mutations":1} 2022-03-07 11:58:41,726 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646654021726","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:32936","multi.service_calls":0,"queuetimems":21344,"class":"HRegionServer","multi.mutations":1} 2022-03-07 11:58:51,353 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646653431352","responsesize":"16531","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 10 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.30.41.118:32804","multi.service_calls":0,"queuetimems":10270,"class":"HRegionServer","multi.mutations":10} 2022-03-07 12:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=319528, hits=311418, hitRatio=97.46%, , cachingAccesses=311548, cachingHits=311109, cachingHitsRatio=99.86%, evictions=20266, evicted=0, evictedPerRun=0.0 2022-03-07 12:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 12:02:18,832 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646653938831","responsesize":"11572","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 7 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:35518","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":7} 2022-03-07 12:02:39,669 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646654259668","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:35550","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:02:40,161 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-07 12:02:48,837 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646654268837","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:32988","multi.service_calls":0,"queuetimems":9169,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:02:55,528 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2228ms No GCs detected 2022-03-07 12:03:03,318 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646654283318","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:35550","multi.service_calls":0,"queuetimems":23650,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:03:09,796 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646653689788","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 8 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900008,"client":"172.30.41.155:35504","multi.service_calls":0,"queuetimems":50148,"class":"HRegionServer","multi.mutations":8} 2022-03-07 12:03:24,022 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646654304020","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.118:32994","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:03:35,658 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646654315657","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:35556","multi.service_calls":0,"queuetimems":11637,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:03:35,658 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646654315657","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:35556","multi.service_calls":0,"queuetimems":11637,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:03:51,354 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646654331353","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:35556","multi.service_calls":0,"queuetimems":27333,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=320133, hits=312023, hitRatio=97.47%, , cachingAccesses=312153, cachingHits=311714, cachingHitsRatio=99.86%, evictions=20296, evicted=0, evictedPerRun=0.0 2022-03-07 12:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 12:07:18,834 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646654538833","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:33024","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:07:19,642 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646653939641","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:32928","multi.service_calls":0,"queuetimems":810,"class":"HRegionServer","multi.mutations":3} 2022-03-07 12:07:25,361 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646653945359","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.118:32928","multi.service_calls":0,"queuetimems":6528,"class":"HRegionServer","multi.mutations":5} 2022-03-07 12:07:39,670 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646654559669","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:33024","multi.service_calls":0,"queuetimems":20836,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:07:58,686 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646653678648","responsesize":"16534","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 10 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200038,"client":"172.30.41.118:32854","multi.service_calls":0,"queuetimems":33289,"class":"HRegionServer","multi.mutations":10} 2022-03-07 12:08:03,319 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646654583318","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:35580","multi.service_calls":0,"queuetimems":44485,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:08:09,797 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646654589796","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:35580","multi.service_calls":0,"queuetimems":30126,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:08:24,023 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646654604023","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:33036","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:08:35,659 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646654615658","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:54814","multi.service_calls":0,"queuetimems":11635,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:08:38,115 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646654318114","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:32994","multi.service_calls":0,"queuetimems":14094,"class":"HRegionServer","multi.mutations":2} 2022-03-07 12:08:39,825 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646654319824","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:35556","multi.service_calls":0,"queuetimems":15804,"class":"HRegionServer","multi.mutations":2} 2022-03-07 12:08:39,953 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646654319952","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:32994","multi.service_calls":0,"queuetimems":4294,"class":"HRegionServer","multi.mutations":4} 2022-03-07 12:09:52,599 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1507ms No GCs detected 2022-03-07 12:10:17,691 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-07 12:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=320947, hits=312837, hitRatio=97.47%, , cachingAccesses=312967, cachingHits=312528, cachingHitsRatio=99.86%, evictions=20326, evicted=0, evictedPerRun=0.0 2022-03-07 12:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 12:12:18,835 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646654838835","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:33080","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 12:12:19,643 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646654839643","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:35602","multi.service_calls":0,"queuetimems":808,"class":"HRegionServer","multi.mutations":3} 2022-03-07 12:12:25,363 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646654845361","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:33080","multi.service_calls":0,"queuetimems":6526,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:12:39,670 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646654859670","responsesize":"6613","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:54828","multi.service_calls":0,"queuetimems":20835,"class":"HRegionServer","multi.mutations":4} 2022-03-07 12:12:48,838 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646654568837","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 5 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:35580","multi.service_calls":0,"queuetimems":30004,"class":"HRegionServer","multi.mutations":5} 2022-03-07 12:13:03,320 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646654883319","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:33096","multi.service_calls":0,"queuetimems":4631,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:13:09,797 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646654889797","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:54830","multi.service_calls":0,"queuetimems":11109,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:13:24,024 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646654904024","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:35604","multi.service_calls":0,"queuetimems":25336,"class":"HRegionServer","multi.mutations":2} 2022-03-07 12:13:35,659 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646654615658","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 8 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:33036","multi.service_calls":0,"queuetimems":11635,"class":"HRegionServer","multi.mutations":8} 2022-03-07 12:13:38,116 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646654918115","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:35604","multi.service_calls":0,"queuetimems":34795,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:13:39,917 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646654919825","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300092,"client":"172.30.41.155:35604","multi.service_calls":0,"queuetimems":36505,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:13:39,954 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646654919953","responsesize":"13221","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 8 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:54830","multi.service_calls":0,"queuetimems":36633,"class":"HRegionServer","multi.mutations":8} 2022-03-07 12:13:41,727 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646654321726","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:32994","multi.service_calls":0,"queuetimems":17706,"class":"HRegionServer","multi.mutations":4} 2022-03-07 12:13:51,355 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646654631354","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:35586","multi.service_calls":0,"queuetimems":27331,"class":"HRegionServer","multi.mutations":3} 2022-03-07 12:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=321636, hits=313526, hitRatio=97.48%, , cachingAccesses=313656, cachingHits=313217, cachingHitsRatio=99.86%, evictions=20356, evicted=0, evictedPerRun=0.0 2022-03-07 12:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 12:17:18,837 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646655138836","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:33162","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:17:19,644 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646655139644","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:33162","multi.service_calls":0,"queuetimems":808,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:17:25,363 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646655145363","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:33162","multi.service_calls":0,"queuetimems":6527,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:17:39,671 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646655159670","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:33162","multi.service_calls":0,"queuetimems":20834,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:17:48,839 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646655168838","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:33162","multi.service_calls":0,"queuetimems":30002,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:17:58,696 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646654878689","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600007,"client":"172.31.0.219:54830","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":4} 2022-03-07 12:18:03,320 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646655183320","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:35620","multi.service_calls":0,"queuetimems":44484,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:18:09,798 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646655189797","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:33162","multi.service_calls":0,"queuetimems":50961,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:18:24,026 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646655204025","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:33172","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:18:35,659 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646655215659","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:33172","multi.service_calls":0,"queuetimems":11634,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:18:35,659 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646654915659","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:35604","multi.service_calls":0,"queuetimems":36970,"class":"HRegionServer","multi.mutations":3} 2022-03-07 12:18:38,117 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646655218116","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:35622","multi.service_calls":0,"queuetimems":14091,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:18:39,918 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646655219917","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:33172","multi.service_calls":0,"queuetimems":15892,"class":"HRegionServer","multi.mutations":2} 2022-03-07 12:18:39,956 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646655219954","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.155:35622","multi.service_calls":0,"queuetimems":15929,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:18:41,728 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646655221728","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:35622","multi.service_calls":0,"queuetimems":17703,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:18:51,355 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646655231355","responsesize":"9917","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:33172","multi.service_calls":0,"queuetimems":27330,"class":"HRegionServer","multi.mutations":6} 2022-03-07 12:20:25,320 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3384ms No GCs detected 2022-03-07 12:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=322465, hits=314355, hitRatio=97.48%, , cachingAccesses=314485, cachingHits=314046, cachingHitsRatio=99.86%, evictions=20386, evicted=0, evictedPerRun=0.0 2022-03-07 12:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 12:22:25,364 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646655445363","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:33214","multi.service_calls":0,"queuetimems":6525,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:22:48,839 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646655468839","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:54872","multi.service_calls":0,"queuetimems":30001,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:22:58,698 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646655478697","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:35634","multi.service_calls":0,"queuetimems":39859,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:23:36,147 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646655515659","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300488,"client":"172.30.41.155:35636","multi.service_calls":0,"queuetimems":11632,"class":"HRegionServer","multi.mutations":2} 2022-03-07 12:23:38,117 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646655518117","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:33224","multi.service_calls":0,"queuetimems":2457,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:23:39,956 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646655519956","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:54884","multi.service_calls":0,"queuetimems":4295,"class":"HRegionServer","multi.mutations":2} 2022-03-07 12:24:10,361 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1324ms No GCs detected 2022-03-07 12:25:57,851 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-07 12:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=323224, hits=315114, hitRatio=97.49%, , cachingAccesses=315244, cachingHits=314805, cachingHitsRatio=99.86%, evictions=20416, evicted=0, evictedPerRun=0.0 2022-03-07 12:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 12:27:25,366 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646655745365","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:33264","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:27:48,840 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646655768839","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:33264","multi.service_calls":0,"queuetimems":23474,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:27:58,701 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646655778701","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:54922","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:28:03,321 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646655483321","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:35634","multi.service_calls":0,"queuetimems":43676,"class":"HRegionServer","multi.mutations":2} 2022-03-07 12:28:09,799 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646655489798","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 5 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:35634","multi.service_calls":0,"queuetimems":50153,"class":"HRegionServer","multi.mutations":5} 2022-03-07 12:28:36,147 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646655816147","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:33266","multi.service_calls":0,"queuetimems":37447,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:28:36,149 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646655515660","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600489,"client":"172.30.41.155:35636","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 12:28:38,118 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646655818118","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:33266","multi.service_calls":0,"queuetimems":39418,"class":"HRegionServer","multi.mutations":2} 2022-03-07 12:28:39,957 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646655819957","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:33266","multi.service_calls":0,"queuetimems":41257,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:28:41,729 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646655521728","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:35636","multi.service_calls":0,"queuetimems":17701,"class":"HRegionServer","multi.mutations":3} 2022-03-07 12:28:51,356 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646655531355","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:54884","multi.service_calls":0,"queuetimems":27327,"class":"HRegionServer","multi.mutations":2} 2022-03-07 12:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=324015, hits=315905, hitRatio=97.50%, , cachingAccesses=316035, cachingHits=315596, cachingHitsRatio=99.86%, evictions=20446, evicted=0, evictedPerRun=0.0 2022-03-07 12:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-07 12:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 12:32:18,839 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646655438838","responsesize":"19835","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 12 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.30.41.118:33214","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":12} 2022-03-07 12:32:19,646 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646655439645","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:35634","multi.service_calls":0,"queuetimems":807,"class":"HRegionServer","multi.mutations":4} 2022-03-07 12:32:25,368 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646656045368","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:54950","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:32:39,672 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646655459671","responsesize":"18183","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 11 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.31.0.219:54872","multi.service_calls":0,"queuetimems":20833,"class":"HRegionServer","multi.mutations":11} 2022-03-07 12:32:58,702 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646656078701","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:35674","multi.service_calls":0,"queuetimems":33333,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:33:03,322 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646656083322","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:54950","multi.service_calls":0,"queuetimems":37954,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:33:09,799 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646656089799","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:54950","multi.service_calls":0,"queuetimems":44431,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:33:36,148 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646656116147","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:54950","multi.service_calls":0,"queuetimems":70779,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:33:36,149 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646656116149","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:54950","multi.service_calls":0,"queuetimems":70781,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:33:38,119 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646656118118","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:54950","multi.service_calls":0,"queuetimems":34796,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:33:39,919 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646655519918","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 5 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:35636","multi.service_calls":0,"queuetimems":4258,"class":"HRegionServer","multi.mutations":5} 2022-03-07 12:33:41,729 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646656121729","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:54950","multi.service_calls":0,"queuetimems":76361,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=324837, hits=316727, hitRatio=97.50%, , cachingAccesses=316857, cachingHits=316418, cachingHitsRatio=99.86%, evictions=20476, evicted=0, evictedPerRun=0.0 2022-03-07 12:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 12:37:18,841 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646656338840","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:33354","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:37:48,841 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646656068840","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 6 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:33312","multi.service_calls":0,"queuetimems":23472,"class":"HRegionServer","multi.mutations":6} 2022-03-07 12:37:58,702 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646656378702","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:54974","multi.service_calls":0,"queuetimems":39055,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:38:03,323 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646656383322","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:54974","multi.service_calls":0,"queuetimems":44481,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:38:09,800 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646656389799","responsesize":"13221","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 8 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:54974","multi.service_calls":0,"queuetimems":50958,"class":"HRegionServer","multi.mutations":8} 2022-03-07 12:38:24,035 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646655504027","responsesize":"19838","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 12 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200008,"client":"172.30.41.118:33224","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":12} 2022-03-07 12:38:36,151 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646656416149","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.118:33374","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:38:36,151 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646656416150","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:54978","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":2} 2022-03-07 12:38:38,119 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646656418119","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:35714","multi.service_calls":0,"queuetimems":1970,"class":"HRegionServer","multi.mutations":2} 2022-03-07 12:38:39,958 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646656119957","responsesize":"16528","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 10 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":600001,"client":"172.31.0.219:54950","multi.service_calls":0,"queuetimems":36634,"class":"HRegionServer","multi.mutations":10} 2022-03-07 12:38:51,358 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646656131357","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:33320","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 12:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=325630, hits=317520, hitRatio=97.51%, , cachingAccesses=317650, cachingHits=317211, cachingHitsRatio=99.86%, evictions=20506, evicted=0, evictedPerRun=0.0 2022-03-07 12:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 12:42:25,370 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646656345368","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.155:35704","multi.service_calls":0,"queuetimems":6527,"class":"HRegionServer","multi.mutations":3} 2022-03-07 12:42:39,673 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646656359672","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 6 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:35704","multi.service_calls":0,"queuetimems":20832,"class":"HRegionServer","multi.mutations":6} 2022-03-07 12:42:48,841 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646656668841","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:35744","multi.service_calls":0,"queuetimems":29999,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:43:03,324 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646656683323","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:55000","multi.service_calls":0,"queuetimems":44481,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:43:09,801 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646656689800","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:35744","multi.service_calls":0,"queuetimems":6476,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:43:24,036 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646656704036","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:55000","multi.service_calls":0,"queuetimems":65194,"class":"HRegionServer","multi.mutations":2} 2022-03-07 12:43:38,120 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646656718119","responsesize":"8265","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 5 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:33422","multi.service_calls":0,"queuetimems":28318,"class":"HRegionServer","multi.mutations":5} 2022-03-07 12:43:39,920 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646656419919","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:35714","multi.service_calls":0,"queuetimems":3769,"class":"HRegionServer","multi.mutations":5} 2022-03-07 12:43:39,959 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646656719958","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:33422","multi.service_calls":0,"queuetimems":3806,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:43:41,730 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646656421730","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:35714","multi.service_calls":0,"queuetimems":5581,"class":"HRegionServer","multi.mutations":2} 2022-03-07 12:43:51,358 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646656731358","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:55000","multi.service_calls":0,"queuetimems":48033,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=326526, hits=318416, hitRatio=97.52%, , cachingAccesses=318546, cachingHits=318107, cachingHitsRatio=99.86%, evictions=20536, evicted=0, evictedPerRun=0.0 2022-03-07 12:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 12:47:18,843 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646656638842","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:33416","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":5} 2022-03-07 12:47:19,648 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646656339647","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:33354","multi.service_calls":0,"queuetimems":807,"class":"HRegionServer","multi.mutations":7} 2022-03-07 12:48:09,851 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646656989850","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300000,"client":"172.30.41.118:33476","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:48:24,038 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646657004036","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.155:35778","multi.service_calls":0,"queuetimems":44362,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:48:36,152 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646656716151","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 9 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:55000","multi.service_calls":0,"queuetimems":32827,"class":"HRegionServer","multi.mutations":9} 2022-03-07 12:48:36,152 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646656716152","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.31.0.219:55000","multi.service_calls":0,"queuetimems":32828,"class":"HRegionServer","multi.mutations":2} 2022-03-07 12:48:39,921 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646657019920","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:55032","multi.service_calls":0,"queuetimems":1799,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:48:39,980 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646657019979","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300001,"client":"172.30.41.118:33480","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=327383, hits=319273, hitRatio=97.52%, , cachingAccesses=319403, cachingHits=318964, cachingHitsRatio=99.86%, evictions=20566, evicted=0, evictedPerRun=0.0 2022-03-07 12:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 12:52:18,844 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646657238843","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:35812","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:52:19,649 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646657239649","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:33526","multi.service_calls":0,"queuetimems":805,"class":"HRegionServer","multi.mutations":3} 2022-03-07 12:52:25,372 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646656945371","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:33468","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 12:53:09,852 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646657289851","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:33536","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:53:24,039 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646657304038","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:33536","multi.service_calls":0,"queuetimems":14187,"class":"HRegionServer","multi.mutations":2} 2022-03-07 12:53:36,153 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646657316152","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:35816","multi.service_calls":0,"queuetimems":26301,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:53:36,153 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646657316152","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:33536","multi.service_calls":0,"queuetimems":26300,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:53:38,122 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646657018121","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:33478","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-07 12:53:39,980 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646657319980","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:35816","multi.service_calls":0,"queuetimems":15941,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=328130, hits=320020, hitRatio=97.53%, , cachingAccesses=320150, cachingHits=319711, cachingHitsRatio=99.86%, evictions=20596, evicted=0, evictedPerRun=0.0 2022-03-07 12:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 12:57:58,710 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646656678702","responsesize":"13230","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 8 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200007,"client":"172.30.41.155:35744","multi.service_calls":0,"queuetimems":39860,"class":"HRegionServer","multi.mutations":8} 2022-03-07 12:58:03,325 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646656983324","responsesize":"18183","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 11 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.31.0.219:55028","multi.service_calls":0,"queuetimems":37952,"class":"HRegionServer","multi.mutations":11} 2022-03-07 12:58:09,853 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646657589852","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:35844","multi.service_calls":0,"queuetimems":50202,"class":"HRegionServer","multi.mutations":2} 2022-03-07 12:58:24,040 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646657604039","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:33602","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:58:36,154 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646657616153","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:33602","multi.service_calls":0,"queuetimems":12114,"class":"HRegionServer","multi.mutations":1} 2022-03-07 12:58:41,732 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646657021731","responsesize":"19835","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 12 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.30.41.118:33478","multi.service_calls":0,"queuetimems":3610,"class":"HRegionServer","multi.mutations":12} 2022-03-07 12:58:51,360 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646657031358","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 7 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:55032","multi.service_calls":0,"queuetimems":13237,"class":"HRegionServer","multi.mutations":7} 2022-03-07 13:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=328936, hits=320826, hitRatio=97.53%, , cachingAccesses=320956, cachingHits=320517, cachingHitsRatio=99.86%, evictions=20626, evicted=0, evictedPerRun=0.0 2022-03-07 13:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 13:02:19,651 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646657539650","responsesize":"16528","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 10 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":600001,"client":"172.31.0.219:55088","multi.service_calls":0,"queuetimems":805,"class":"HRegionServer","multi.mutations":10} 2022-03-07 13:02:48,843 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646656968841","responsesize":"16534","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 10 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.155:35778","multi.service_calls":0,"queuetimems":23470,"class":"HRegionServer","multi.mutations":10} 2022-03-07 13:02:58,711 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646657878710","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:33672","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 13:03:03,330 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646657883326","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300004,"client":"172.30.41.118:33672","multi.service_calls":0,"queuetimems":4615,"class":"HRegionServer","multi.mutations":1} 2022-03-07 13:03:09,853 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646657889853","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:33672","multi.service_calls":0,"queuetimems":6527,"class":"HRegionServer","multi.mutations":2} 2022-03-07 13:03:24,041 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646657904040","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:33672","multi.service_calls":0,"queuetimems":25329,"class":"HRegionServer","multi.mutations":1} 2022-03-07 13:03:38,123 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646657618122","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:35846","multi.service_calls":0,"queuetimems":14082,"class":"HRegionServer","multi.mutations":4} 2022-03-07 13:03:39,922 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646657319921","responsesize":"19835","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 12 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.30.41.118:33536","multi.service_calls":0,"queuetimems":15882,"class":"HRegionServer","multi.mutations":12} 2022-03-07 13:03:39,981 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646657619980","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:55090","multi.service_calls":0,"queuetimems":15940,"class":"HRegionServer","multi.mutations":3} 2022-03-07 13:03:41,733 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646657921732","responsesize":"6613","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:55108","multi.service_calls":0,"queuetimems":43021,"class":"HRegionServer","multi.mutations":4} 2022-03-07 13:03:51,360 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646657931360","responsesize":"6613","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:55108","multi.service_calls":0,"queuetimems":52649,"class":"HRegionServer","multi.mutations":4} 2022-03-07 13:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=329678, hits=321568, hitRatio=97.54%, , cachingAccesses=321698, cachingHits=321259, cachingHitsRatio=99.86%, evictions=20656, evicted=0, evictedPerRun=0.0 2022-03-07 13:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 13:07:19,652 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646658139652","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:33728","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 13:07:25,374 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646657545372","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.31.0.219:55088","multi.service_calls":0,"queuetimems":6527,"class":"HRegionServer","multi.mutations":4} 2022-03-07 13:07:39,676 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646656959674","responsesize":"28101","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 17 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1500002,"client":"172.30.41.118:33468","multi.service_calls":0,"queuetimems":14303,"class":"HRegionServer","multi.mutations":17} 2022-03-07 13:07:48,844 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646658168844","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:55126","multi.service_calls":0,"queuetimems":29192,"class":"HRegionServer","multi.mutations":1} 2022-03-07 13:07:58,713 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646658178711","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.118:33728","multi.service_calls":0,"queuetimems":9867,"class":"HRegionServer","multi.mutations":1} 2022-03-07 13:08:24,042 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646658204042","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:35876","multi.service_calls":0,"queuetimems":35198,"class":"HRegionServer","multi.mutations":1} 2022-03-07 13:08:36,154 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646657616153","responsesize":"16531","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 10 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.30.41.118:33602","multi.service_calls":0,"queuetimems":12114,"class":"HRegionServer","multi.mutations":10} 2022-03-07 13:08:38,124 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646658218123","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:33752","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 13:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=330420, hits=322310, hitRatio=97.55%, , cachingAccesses=322440, cachingHits=322001, cachingHitsRatio=99.86%, evictions=20686, evicted=0, evictedPerRun=0.0 2022-03-07 13:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 13:12:18,847 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646657538845","responsesize":"16534","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 10 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.155:35844","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":10} 2022-03-07 13:12:19,654 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646658439653","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:35894","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 13:12:25,376 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646658445375","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:33804","multi.service_calls":0,"queuetimems":5721,"class":"HRegionServer","multi.mutations":2} 2022-03-07 13:12:39,676 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646658459676","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300000,"client":"172.30.41.118:33804","multi.service_calls":0,"queuetimems":20023,"class":"HRegionServer","multi.mutations":2} 2022-03-07 13:12:48,845 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646658468844","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:35894","multi.service_calls":0,"queuetimems":29190,"class":"HRegionServer","multi.mutations":1} 2022-03-07 13:12:58,714 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646658478713","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:35894","multi.service_calls":0,"queuetimems":39059,"class":"HRegionServer","multi.mutations":1} 2022-03-07 13:13:03,331 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646658183330","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:35876","multi.service_calls":0,"queuetimems":43678,"class":"HRegionServer","multi.mutations":3} 2022-03-07 13:13:39,982 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646658219981","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:55128","multi.service_calls":0,"queuetimems":1857,"class":"HRegionServer","multi.mutations":2} 2022-03-07 13:13:41,734 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646658221733","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:35878","multi.service_calls":0,"queuetimems":3610,"class":"HRegionServer","multi.mutations":2} 2022-03-07 13:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=331204, hits=323094, hitRatio=97.55%, , cachingAccesses=323224, cachingHits=322785, cachingHitsRatio=99.86%, evictions=20716, evicted=0, evictedPerRun=0.0 2022-03-07 13:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 13:17:18,849 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646658738848","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:35912","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 13:17:39,677 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646658759677","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:33878","multi.service_calls":0,"queuetimems":20022,"class":"HRegionServer","multi.mutations":1} 2022-03-07 13:18:03,331 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646658783331","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300000,"client":"172.30.41.118:33870","multi.service_calls":0,"queuetimems":23625,"class":"HRegionServer","multi.mutations":1} 2022-03-07 13:18:09,855 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646658189853","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.118:33728","multi.service_calls":0,"queuetimems":50201,"class":"HRegionServer","multi.mutations":7} 2022-03-07 13:18:24,045 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646658504043","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.118:33820","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":5} 2022-03-07 13:18:36,155 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646658516154","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 4 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:33820","multi.service_calls":0,"queuetimems":12111,"class":"HRegionServer","multi.mutations":4} 2022-03-07 13:18:36,155 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646657916154","responsesize":"14882","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 9 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200001,"client":"172.31.0.219:55108","multi.service_calls":0,"queuetimems":32827,"class":"HRegionServer","multi.mutations":9} 2022-03-07 13:18:38,125 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646658518124","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:35896","multi.service_calls":0,"queuetimems":14081,"class":"HRegionServer","multi.mutations":2} 2022-03-07 13:18:39,923 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646658219922","responsesize":"18183","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 11 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.30.41.118:33752","multi.service_calls":0,"queuetimems":1798,"class":"HRegionServer","multi.mutations":11} 2022-03-07 13:18:41,735 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646658821735","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:55164","multi.service_calls":0,"queuetimems":1752,"class":"HRegionServer","multi.mutations":1} 2022-03-07 13:18:51,362 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646658231360","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:35878","multi.service_calls":0,"queuetimems":13236,"class":"HRegionServer","multi.mutations":7} 2022-03-07 13:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=331930, hits=323820, hitRatio=97.56%, , cachingAccesses=323950, cachingHits=323511, cachingHitsRatio=99.86%, evictions=20746, evicted=0, evictedPerRun=0.0 2022-03-07 13:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 13:22:39,678 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646659059678","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:55180","multi.service_calls":0,"queuetimems":20828,"class":"HRegionServer","multi.mutations":1} 2022-03-07 13:22:48,846 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646658768845","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 5 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:35912","multi.service_calls":0,"queuetimems":29997,"class":"HRegionServer","multi.mutations":5} 2022-03-07 13:23:03,332 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646659083331","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:55180","multi.service_calls":0,"queuetimems":41978,"class":"HRegionServer","multi.mutations":1} 2022-03-07 13:23:09,855 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646659089855","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:55180","multi.service_calls":0,"queuetimems":48291,"class":"HRegionServer","multi.mutations":1} 2022-03-07 13:23:24,046 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646659104045","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:35932","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 13:23:36,156 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646659116155","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:35932","multi.service_calls":0,"queuetimems":12109,"class":"HRegionServer","multi.mutations":2} 2022-03-07 13:23:39,924 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646659119924","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:55182","multi.service_calls":0,"queuetimems":15878,"class":"HRegionServer","multi.mutations":1} 2022-03-07 13:23:39,984 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646658819983","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:33890","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 13:23:41,736 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646659121735","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:55182","multi.service_calls":0,"queuetimems":17689,"class":"HRegionServer","multi.mutations":1} 2022-03-07 13:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=332648, hits=324538, hitRatio=97.56%, , cachingAccesses=324668, cachingHits=324229, cachingHitsRatio=99.86%, evictions=20776, evicted=0, evictedPerRun=0.0 2022-03-07 13:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 13:27:18,851 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646659038850","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:35930","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 13:27:19,656 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646658739655","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:55162","multi.service_calls":0,"queuetimems":807,"class":"HRegionServer","multi.mutations":3} 2022-03-07 13:27:25,379 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646658745377","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 8 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.118:33878","multi.service_calls":0,"queuetimems":6529,"class":"HRegionServer","multi.mutations":8} 2022-03-07 13:28:03,332 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646659383332","responsesize":"13221","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 8 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:55196","multi.service_calls":0,"queuetimems":23652,"class":"HRegionServer","multi.mutations":8} 2022-03-07 13:28:09,856 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646659389855","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:34004","multi.service_calls":0,"queuetimems":21008,"class":"HRegionServer","multi.mutations":1} 2022-03-07 13:28:24,047 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646659404047","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:34022","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 13:28:39,925 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646659419924","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:35950","multi.service_calls":0,"queuetimems":15877,"class":"HRegionServer","multi.mutations":2} 2022-03-07 13:28:41,736 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646659421736","responsesize":"13221","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 8 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:55198","multi.service_calls":0,"queuetimems":17688,"class":"HRegionServer","multi.mutations":8} 2022-03-07 13:28:51,363 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646659131362","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:35932","multi.service_calls":0,"queuetimems":15206,"class":"HRegionServer","multi.mutations":2} 2022-03-07 13:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=333346, hits=325236, hitRatio=97.57%, , cachingAccesses=325366, cachingHits=324927, cachingHitsRatio=99.87%, evictions=20806, evicted=0, evictedPerRun=0.0 2022-03-07 13:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-07 13:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 13:32:18,852 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646659638852","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:34080","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 13:32:19,657 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646659639656","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:55212","multi.service_calls":0,"queuetimems":804,"class":"HRegionServer","multi.mutations":2} 2022-03-07 13:32:25,380 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646659645379","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:55212","multi.service_calls":0,"queuetimems":6527,"class":"HRegionServer","multi.mutations":1} 2022-03-07 13:32:39,680 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646659359679","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:34004","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 13:32:48,847 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646659368846","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:34004","multi.service_calls":0,"queuetimems":9167,"class":"HRegionServer","multi.mutations":3} 2022-03-07 13:32:58,720 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646658778715","responsesize":"18186","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 11 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200005,"client":"172.31.0.219:55162","multi.service_calls":0,"queuetimems":39867,"class":"HRegionServer","multi.mutations":11} 2022-03-07 13:33:03,333 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646659683333","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:34080","multi.service_calls":0,"queuetimems":43676,"class":"HRegionServer","multi.mutations":2} 2022-03-07 13:33:09,856 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646659689856","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:35964","multi.service_calls":0,"queuetimems":50199,"class":"HRegionServer","multi.mutations":3} 2022-03-07 13:33:36,157 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646659116156","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:33954","multi.service_calls":0,"queuetimems":12111,"class":"HRegionServer","multi.mutations":3} 2022-03-07 13:33:36,157 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646659416156","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:35950","multi.service_calls":0,"queuetimems":12109,"class":"HRegionServer","multi.mutations":3} 2022-03-07 13:33:38,126 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646659118125","responsesize":"18183","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 11 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.30.41.118:33954","multi.service_calls":0,"queuetimems":14079,"class":"HRegionServer","multi.mutations":11} 2022-03-07 13:33:51,363 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646659731363","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:55212","multi.service_calls":0,"queuetimems":91706,"class":"HRegionServer","multi.mutations":1} 2022-03-07 13:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=334128, hits=326018, hitRatio=97.57%, , cachingAccesses=326148, cachingHits=325709, cachingHitsRatio=99.87%, evictions=20836, evicted=0, evictedPerRun=0.0 2022-03-07 13:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 13:37:18,855 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646659938853","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:34128","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 13:37:19,658 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646659939657","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:35998","multi.service_calls":0,"queuetimems":803,"class":"HRegionServer","multi.mutations":1} 2022-03-07 13:37:25,381 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646659945380","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:34128","multi.service_calls":0,"queuetimems":6527,"class":"HRegionServer","multi.mutations":1} 2022-03-07 13:37:48,848 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646659968847","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:55228","multi.service_calls":0,"queuetimems":29993,"class":"HRegionServer","multi.mutations":1} 2022-03-07 13:37:58,723 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646659978722","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:36004","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 13:38:03,334 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646659983333","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:55230","multi.service_calls":0,"queuetimems":4611,"class":"HRegionServer","multi.mutations":1} 2022-03-07 13:38:24,050 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646659704048","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.155:35964","multi.service_calls":0,"queuetimems":65196,"class":"HRegionServer","multi.mutations":2} 2022-03-07 13:38:36,157 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646660016157","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:34130","multi.service_calls":0,"queuetimems":32823,"class":"HRegionServer","multi.mutations":1} 2022-03-07 13:38:36,157 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646660016157","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:34130","multi.service_calls":0,"queuetimems":37435,"class":"HRegionServer","multi.mutations":1} 2022-03-07 13:38:39,985 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646659419984","responsesize":"19835","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 12 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.30.41.118:34022","multi.service_calls":0,"queuetimems":15937,"class":"HRegionServer","multi.mutations":12} 2022-03-07 13:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=334771, hits=326661, hitRatio=97.58%, , cachingAccesses=326791, cachingHits=326352, cachingHitsRatio=99.87%, evictions=20866, evicted=0, evictedPerRun=0.0 2022-03-07 13:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 13:42:18,856 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646660238856","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:36034","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":3} 2022-03-07 13:42:25,382 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646660245381","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:34180","multi.service_calls":0,"queuetimems":6525,"class":"HRegionServer","multi.mutations":2} 2022-03-07 13:42:48,849 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646660268848","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:34180","multi.service_calls":0,"queuetimems":29189,"class":"HRegionServer","multi.mutations":1} 2022-03-07 13:43:09,858 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646659989857","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 9 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:34130","multi.service_calls":0,"queuetimems":11135,"class":"HRegionServer","multi.mutations":9} 2022-03-07 13:43:11,267 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646660291267","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:34072","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 13:43:36,158 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646660316157","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:36040","multi.service_calls":0,"queuetimems":11063,"class":"HRegionServer","multi.mutations":1} 2022-03-07 13:43:36,158 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646660316157","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:36040","multi.service_calls":0,"queuetimems":23008,"class":"HRegionServer","multi.mutations":1} 2022-03-07 13:43:39,926 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646659719925","responsesize":"19835","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 12 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.31.0.219:55212","multi.service_calls":0,"queuetimems":80268,"class":"HRegionServer","multi.mutations":12} 2022-03-07 13:43:39,986 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646660319985","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:55248","multi.service_calls":0,"queuetimems":19636,"class":"HRegionServer","multi.mutations":1} 2022-03-07 13:43:51,364 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646660031363","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 5 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:55230","multi.service_calls":0,"queuetimems":52641,"class":"HRegionServer","multi.mutations":5} 2022-03-07 13:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=335474, hits=327364, hitRatio=97.58%, , cachingAccesses=327494, cachingHits=327055, cachingHitsRatio=99.87%, evictions=20896, evicted=0, evictedPerRun=0.0 2022-03-07 13:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 13:47:18,857 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646660538857","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:34236","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 13:47:19,659 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646660239658","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 4 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:55244","multi.service_calls":0,"queuetimems":802,"class":"HRegionServer","multi.mutations":4} 2022-03-07 13:47:25,383 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646660545382","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:34236","multi.service_calls":0,"queuetimems":6525,"class":"HRegionServer","multi.mutations":3} 2022-03-07 13:48:09,859 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646660589858","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:36054","multi.service_calls":0,"queuetimems":51001,"class":"HRegionServer","multi.mutations":3} 2022-03-07 13:48:24,052 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646660304050","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.118:34186","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 13:48:36,158 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646660616158","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:36054","multi.service_calls":0,"queuetimems":26300,"class":"HRegionServer","multi.mutations":1} 2022-03-07 13:48:41,738 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646659721736","responsesize":"21490","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 13 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.118:34080","multi.service_calls":0,"queuetimems":82884,"class":"HRegionServer","multi.mutations":13} 2022-03-07 13:48:51,365 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646660631364","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:55260","multi.service_calls":0,"queuetimems":41505,"class":"HRegionServer","multi.mutations":1} 2022-03-07 13:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=336183, hits=328073, hitRatio=97.59%, , cachingAccesses=328203, cachingHits=327764, cachingHitsRatio=99.87%, evictions=20926, evicted=0, evictedPerRun=0.0 2022-03-07 13:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 13:52:18,859 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646660838858","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:34296","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 13:52:19,660 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646660839660","responsesize":"9917","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:34296","multi.service_calls":0,"queuetimems":802,"class":"HRegionServer","multi.mutations":6} 2022-03-07 13:52:39,682 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646659959680","responsesize":"26446","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 16 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.118:34128","multi.service_calls":0,"queuetimems":20022,"class":"HRegionServer","multi.mutations":16} 2022-03-07 13:52:48,850 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646660568849","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 6 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:36054","multi.service_calls":0,"queuetimems":29992,"class":"HRegionServer","multi.mutations":6} 2022-03-07 13:53:11,268 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646660591268","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 4 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.31.0.219:55260","multi.service_calls":0,"queuetimems":52410,"class":"HRegionServer","multi.mutations":4} 2022-03-07 13:53:24,053 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646660904053","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:55280","multi.service_calls":0,"queuetimems":14193,"class":"HRegionServer","multi.mutations":1} 2022-03-07 13:53:36,159 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646660616158","responsesize":"11572","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:55260","multi.service_calls":0,"queuetimems":77300,"class":"HRegionServer","multi.mutations":7} 2022-03-07 13:53:36,159 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646660916158","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:55280","multi.service_calls":0,"queuetimems":26298,"class":"HRegionServer","multi.mutations":1} 2022-03-07 13:53:38,128 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646660018127","responsesize":"13230","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 8 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200001,"client":"172.30.41.155:36004","multi.service_calls":0,"queuetimems":34793,"class":"HRegionServer","multi.mutations":8} 2022-03-07 13:53:39,927 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646660619926","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:36054","multi.service_calls":0,"queuetimems":30068,"class":"HRegionServer","multi.mutations":2} 2022-03-07 13:53:41,739 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646660921738","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:55280","multi.service_calls":0,"queuetimems":31878,"class":"HRegionServer","multi.mutations":1} 2022-03-07 13:53:51,365 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646660931365","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:55280","multi.service_calls":0,"queuetimems":41505,"class":"HRegionServer","multi.mutations":1} 2022-03-07 13:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=336982, hits=328872, hitRatio=97.59%, , cachingAccesses=329002, cachingHits=328563, cachingHitsRatio=99.87%, evictions=20956, evicted=0, evictedPerRun=0.0 2022-03-07 13:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 13:57:39,683 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646661159682","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:34370","multi.service_calls":0,"queuetimems":20021,"class":"HRegionServer","multi.mutations":1} 2022-03-07 13:58:03,336 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646660283334","responsesize":"26446","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 16 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Dev ","processingtimems":1200002,"client":"172.30.41.118:34180","multi.service_calls":0,"queuetimems":43675,"class":"HRegionServer","multi.mutations":16} 2022-03-07 13:58:09,861 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646660889860","responsesize":"11572","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 7 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:34306","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":7} 2022-03-07 13:58:38,131 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646661218128","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300003,"client":"172.31.0.219:55298","multi.service_calls":0,"queuetimems":14074,"class":"HRegionServer","multi.mutations":1} 2022-03-07 13:58:39,987 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646660619986","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 5 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:55260","multi.service_calls":0,"queuetimems":30127,"class":"HRegionServer","multi.mutations":5} 2022-03-07 13:58:41,739 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646661221739","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:55298","multi.service_calls":0,"queuetimems":5579,"class":"HRegionServer","multi.mutations":1} 2022-03-07 13:58:51,366 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646661231365","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:55298","multi.service_calls":0,"queuetimems":15205,"class":"HRegionServer","multi.mutations":1} 2022-03-07 14:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=337682, hits=329572, hitRatio=97.60%, , cachingAccesses=329702, cachingHits=329263, cachingHitsRatio=99.87%, evictions=20986, evicted=0, evictedPerRun=0.0 2022-03-07 14:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 14:02:19,662 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646661139661","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 6 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:34370","multi.service_calls":0,"queuetimems":801,"class":"HRegionServer","multi.mutations":6} 2022-03-07 14:02:39,684 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646661459683","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:34418","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 14:03:11,269 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646661191268","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:36098","multi.service_calls":0,"queuetimems":51607,"class":"HRegionServer","multi.mutations":2} 2022-03-07 14:03:24,056 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646661204054","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 6 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.155:36104","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":6} 2022-03-07 14:03:36,160 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646661216159","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:36104","multi.service_calls":0,"queuetimems":12105,"class":"HRegionServer","multi.mutations":3} 2022-03-07 14:03:36,160 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646661216159","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:34378","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 14:03:38,133 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646661518132","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:36142","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 14:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=338382, hits=330272, hitRatio=97.60%, , cachingAccesses=330402, cachingHits=329963, cachingHitsRatio=99.87%, evictions=21016, evicted=0, evictedPerRun=0.0 2022-03-07 14:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 14:07:18,861 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646661138860","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:36098","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":7} 2022-03-07 14:07:19,663 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646661739663","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:34464","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 14:07:48,851 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646661168850","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 6 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:55294","multi.service_calls":0,"queuetimems":29990,"class":"HRegionServer","multi.mutations":6} 2022-03-07 14:08:03,337 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646661483336","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 5 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:36134","multi.service_calls":0,"queuetimems":23652,"class":"HRegionServer","multi.mutations":5} 2022-03-07 14:08:11,270 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646661791270","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:55342","multi.service_calls":0,"queuetimems":51606,"class":"HRegionServer","multi.mutations":1} 2022-03-07 14:08:24,057 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646661804056","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:55342","multi.service_calls":0,"queuetimems":12785,"class":"HRegionServer","multi.mutations":1} 2022-03-07 14:08:36,161 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646661816160","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:55342","multi.service_calls":0,"queuetimems":76497,"class":"HRegionServer","multi.mutations":1} 2022-03-07 14:08:38,133 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646661818133","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:55342","multi.service_calls":0,"queuetimems":78470,"class":"HRegionServer","multi.mutations":1} 2022-03-07 14:08:39,929 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646661219927","responsesize":"16531","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 10 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900002,"client":"172.30.41.118:34378","multi.service_calls":0,"queuetimems":3767,"class":"HRegionServer","multi.mutations":10} 2022-03-07 14:08:39,988 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646661519987","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 9 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:34430","multi.service_calls":0,"queuetimems":1855,"class":"HRegionServer","multi.mutations":9} 2022-03-07 14:08:41,740 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646661521740","responsesize":"11572","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 7 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:36142","multi.service_calls":0,"queuetimems":3608,"class":"HRegionServer","multi.mutations":7} 2022-03-07 14:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=339117, hits=331007, hitRatio=97.61%, , cachingAccesses=331137, cachingHits=330698, cachingHitsRatio=99.87%, evictions=21046, evicted=0, evictedPerRun=0.0 2022-03-07 14:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 14:12:18,863 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646662038862","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:36168","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 14:12:19,664 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646662039664","responsesize":"6613","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:34514","multi.service_calls":0,"queuetimems":802,"class":"HRegionServer","multi.mutations":4} 2022-03-07 14:12:25,392 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646660845383","responsesize":"23145","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 14 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1500008,"client":"172.31.0.219:55278","multi.service_calls":0,"queuetimems":6524,"class":"HRegionServer","multi.mutations":14} 2022-03-07 14:12:39,685 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646661759684","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 9 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:34464","multi.service_calls":0,"queuetimems":20021,"class":"HRegionServer","multi.mutations":9} 2022-03-07 14:13:11,271 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646662091270","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:34514","multi.service_calls":0,"queuetimems":51606,"class":"HRegionServer","multi.mutations":1} 2022-03-07 14:13:24,058 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646662104058","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:34526","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 14:13:36,161 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646661816160","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 5 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:36158","multi.service_calls":0,"queuetimems":12103,"class":"HRegionServer","multi.mutations":5} 2022-03-07 14:13:39,929 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646662119929","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:36170","multi.service_calls":0,"queuetimems":3767,"class":"HRegionServer","multi.mutations":1} 2022-03-07 14:13:51,368 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646661531366","responsesize":"19835","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 12 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.31.0.219:55318","multi.service_calls":0,"queuetimems":13233,"class":"HRegionServer","multi.mutations":12} 2022-03-07 14:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=339869, hits=331759, hitRatio=97.61%, , cachingAccesses=331889, cachingHits=331450, cachingHitsRatio=99.87%, evictions=21076, evicted=0, evictedPerRun=0.0 2022-03-07 14:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 14:17:19,666 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646662339665","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:36176","multi.service_calls":0,"queuetimems":801,"class":"HRegionServer","multi.mutations":3} 2022-03-07 14:17:39,686 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646662359685","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:34578","multi.service_calls":0,"queuetimems":20020,"class":"HRegionServer","multi.mutations":1} 2022-03-07 14:17:48,852 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646662068851","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 9 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:55376","multi.service_calls":0,"queuetimems":29988,"class":"HRegionServer","multi.mutations":9} 2022-03-07 14:18:03,338 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646662083337","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:36168","multi.service_calls":0,"queuetimems":44475,"class":"HRegionServer","multi.mutations":4} 2022-03-07 14:18:09,862 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646661489861","responsesize":"16534","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 10 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200001,"client":"172.31.0.219:55316","multi.service_calls":0,"queuetimems":30177,"class":"HRegionServer","multi.mutations":10} 2022-03-07 14:18:36,162 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646662416161","responsesize":"8265","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:55414","multi.service_calls":0,"queuetimems":24889,"class":"HRegionServer","multi.mutations":5} 2022-03-07 14:18:36,162 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646662116161","responsesize":"16528","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 10 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":600001,"client":"172.30.41.118:34526","multi.service_calls":0,"queuetimems":12103,"class":"HRegionServer","multi.mutations":10} 2022-03-07 14:18:38,134 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646662118133","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 6 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:36170","multi.service_calls":0,"queuetimems":14075,"class":"HRegionServer","multi.mutations":6} 2022-03-07 14:18:39,930 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646662419930","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:55414","multi.service_calls":0,"queuetimems":28658,"class":"HRegionServer","multi.mutations":3} 2022-03-07 14:18:39,989 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646662119989","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:34526","multi.service_calls":0,"queuetimems":3827,"class":"HRegionServer","multi.mutations":2} 2022-03-07 14:18:41,741 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646662121741","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:36170","multi.service_calls":0,"queuetimems":17683,"class":"HRegionServer","multi.mutations":2} 2022-03-07 14:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=340655, hits=332545, hitRatio=97.62%, , cachingAccesses=332675, cachingHits=332236, cachingHitsRatio=99.87%, evictions=21106, evicted=0, evictedPerRun=0.0 2022-03-07 14:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 14:22:18,865 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646662338864","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:36176","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 14:22:19,667 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646662639666","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:34650","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 14:22:25,393 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646662345392","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 6 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:34578","multi.service_calls":0,"queuetimems":6528,"class":"HRegionServer","multi.mutations":6} 2022-03-07 14:22:39,687 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646662659686","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:34650","multi.service_calls":0,"queuetimems":20019,"class":"HRegionServer","multi.mutations":2} 2022-03-07 14:23:09,863 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646662689862","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:34650","multi.service_calls":0,"queuetimems":21009,"class":"HRegionServer","multi.mutations":1} 2022-03-07 14:23:11,273 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646662391272","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 5 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:36178","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":5} 2022-03-07 14:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=341468, hits=333358, hitRatio=97.62%, , cachingAccesses=333488, cachingHits=333049, cachingHitsRatio=99.87%, evictions=21136, evicted=0, evictedPerRun=0.0 2022-03-07 14:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 14:27:19,668 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646662939667","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300001,"client":"172.30.41.118:34718","multi.service_calls":0,"queuetimems":801,"class":"HRegionServer","multi.mutations":1} 2022-03-07 14:27:48,854 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646662668853","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 8 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:55430","multi.service_calls":0,"queuetimems":29186,"class":"HRegionServer","multi.mutations":8} 2022-03-07 14:28:03,339 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646662683338","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:55430","multi.service_calls":0,"queuetimems":43671,"class":"HRegionServer","multi.mutations":3} 2022-03-07 14:28:09,864 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646662989864","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:34728","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 14:28:38,135 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646662718134","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:36198","multi.service_calls":0,"queuetimems":1971,"class":"HRegionServer","multi.mutations":2} 2022-03-07 14:28:41,742 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646662721741","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:55432","multi.service_calls":0,"queuetimems":5578,"class":"HRegionServer","multi.mutations":4} 2022-03-07 14:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=342255, hits=334145, hitRatio=97.63%, , cachingAccesses=334275, cachingHits=333836, cachingHitsRatio=99.87%, evictions=21166, evicted=0, evictedPerRun=0.0 2022-03-07 14:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-07 14:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 14:32:18,867 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646662938866","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:36214","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 14:32:25,395 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646662945393","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.155:36214","multi.service_calls":0,"queuetimems":6527,"class":"HRegionServer","multi.mutations":3} 2022-03-07 14:32:39,688 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646662959687","responsesize":"16528","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 10 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":600001,"client":"172.30.41.118:34718","multi.service_calls":0,"queuetimems":20821,"class":"HRegionServer","multi.mutations":10} 2022-03-07 14:33:24,062 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646662404058","responsesize":"11578","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 7 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200004,"client":"172.30.41.155:36178","multi.service_calls":0,"queuetimems":12786,"class":"HRegionServer","multi.mutations":7} 2022-03-07 14:33:38,136 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646663318136","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:34802","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 14:33:39,991 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646662719990","responsesize":"21487","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 13 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.31.0.219:55432","multi.service_calls":0,"queuetimems":3827,"class":"HRegionServer","multi.mutations":13} 2022-03-07 14:33:51,369 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646662431368","responsesize":"13230","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 8 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200001,"client":"172.31.0.219:55414","multi.service_calls":0,"queuetimems":11438,"class":"HRegionServer","multi.mutations":8} 2022-03-07 14:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=343020, hits=334910, hitRatio=97.64%, , cachingAccesses=335040, cachingHits=334601, cachingHitsRatio=99.87%, evictions=21196, evicted=0, evictedPerRun=0.0 2022-03-07 14:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 14:37:18,869 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646663538868","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:36254","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 14:37:19,670 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646663239669","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:36238","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 14:37:48,855 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646663268854","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 5 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:34788","multi.service_calls":0,"queuetimems":29185,"class":"HRegionServer","multi.mutations":5} 2022-03-07 14:38:03,340 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646663283339","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 5 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:34788","multi.service_calls":0,"queuetimems":43670,"class":"HRegionServer","multi.mutations":5} 2022-03-07 14:38:09,865 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646663289864","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 9 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:55464","multi.service_calls":0,"queuetimems":50195,"class":"HRegionServer","multi.mutations":9} 2022-03-07 14:38:11,274 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646662991273","responsesize":"18183","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 11 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.30.41.118:34728","multi.service_calls":0,"queuetimems":1409,"class":"HRegionServer","multi.mutations":11} 2022-03-07 14:38:24,063 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646663604063","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:34868","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 14:38:36,164 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646662716163","responsesize":"9926","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200001,"client":"172.30.41.118:34666","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":6} 2022-03-07 14:38:39,992 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646663619991","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:55490","multi.service_calls":0,"queuetimems":15928,"class":"HRegionServer","multi.mutations":1} 2022-03-07 14:38:41,743 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646663321743","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:34802","multi.service_calls":0,"queuetimems":3607,"class":"HRegionServer","multi.mutations":3} 2022-03-07 14:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=343811, hits=335701, hitRatio=97.64%, , cachingAccesses=335831, cachingHits=335392, cachingHitsRatio=99.87%, evictions=21226, evicted=0, evictedPerRun=0.0 2022-03-07 14:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 14:42:18,871 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646663838870","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:34914","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 14:42:39,690 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646663559689","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 6 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:36254","multi.service_calls":0,"queuetimems":20821,"class":"HRegionServer","multi.mutations":6} 2022-03-07 14:42:48,856 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646663868855","responsesize":"8265","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:55520","multi.service_calls":0,"queuetimems":29985,"class":"HRegionServer","multi.mutations":5} 2022-03-07 14:43:09,866 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646663889865","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:34914","multi.service_calls":0,"queuetimems":50195,"class":"HRegionServer","multi.mutations":1} 2022-03-07 14:43:24,075 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646663904073","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:34920","multi.service_calls":0,"queuetimems":9,"class":"HRegionServer","multi.mutations":2} 2022-03-07 14:43:36,165 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646662716163","responsesize":"21493","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 13 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1500001,"client":"172.30.41.118:34666","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":13} 2022-03-07 14:43:36,166 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646663916165","responsesize":"6613","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:36272","multi.service_calls":0,"queuetimems":12092,"class":"HRegionServer","multi.mutations":4} 2022-03-07 14:43:39,932 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646662719930","responsesize":"16537","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 10 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1500002,"client":"172.30.41.155:36198","multi.service_calls":0,"queuetimems":3767,"class":"HRegionServer","multi.mutations":10} 2022-03-07 14:43:39,992 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646663919992","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:55526","multi.service_calls":0,"queuetimems":15919,"class":"HRegionServer","multi.mutations":2} 2022-03-07 14:43:51,370 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646663631369","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:36256","multi.service_calls":0,"queuetimems":27306,"class":"HRegionServer","multi.mutations":2} 2022-03-07 14:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=344594, hits=336484, hitRatio=97.65%, , cachingAccesses=336614, cachingHits=336175, cachingHitsRatio=99.87%, evictions=21256, evicted=0, evictedPerRun=0.0 2022-03-07 14:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 14:47:18,872 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646664138872","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:36292","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 14:47:25,397 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646663545395","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":900002,"client":"172.30.41.118:34860","multi.service_calls":0,"queuetimems":6527,"class":"HRegionServer","multi.mutations":4} 2022-03-07 14:48:36,166 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646664216166","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300000,"client":"172.30.41.118:34978","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 14:48:36,166 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646664216166","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:36300","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 14:48:38,138 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646663618137","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:34868","multi.service_calls":0,"queuetimems":14074,"class":"HRegionServer","multi.mutations":7} 2022-03-07 14:48:39,993 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646664219992","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:55564","multi.service_calls":0,"queuetimems":3826,"class":"HRegionServer","multi.mutations":1} 2022-03-07 14:48:41,745 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646663921744","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 8 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:34920","multi.service_calls":0,"queuetimems":17670,"class":"HRegionServer","multi.mutations":8} 2022-03-07 14:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=345365, hits=337255, hitRatio=97.65%, , cachingAccesses=337385, cachingHits=336946, cachingHitsRatio=99.87%, evictions=21286, evicted=0, evictedPerRun=0.0 2022-03-07 14:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 14:52:18,874 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646664438874","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:36326","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 14:52:25,398 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646664445397","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:55578","multi.service_calls":0,"queuetimems":6523,"class":"HRegionServer","multi.mutations":1} 2022-03-07 14:52:39,691 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646664159690","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 5 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:55558","multi.service_calls":0,"queuetimems":20818,"class":"HRegionServer","multi.mutations":5} 2022-03-07 14:52:48,858 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646664168856","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:34970","multi.service_calls":0,"queuetimems":29984,"class":"HRegionServer","multi.mutations":2} 2022-03-07 14:53:03,342 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646663883340","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 5 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:36270","multi.service_calls":0,"queuetimems":44470,"class":"HRegionServer","multi.mutations":5} 2022-03-07 14:53:09,867 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646664189866","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 6 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":600001,"client":"172.30.41.118:34970","multi.service_calls":0,"queuetimems":30175,"class":"HRegionServer","multi.mutations":6} 2022-03-07 14:53:11,276 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646663891274","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.31.0.219:55520","multi.service_calls":0,"queuetimems":52404,"class":"HRegionServer","multi.mutations":7} 2022-03-07 14:53:36,167 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646664516167","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:36336","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 14:53:38,139 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646664518138","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:35026","multi.service_calls":0,"queuetimems":1971,"class":"HRegionServer","multi.mutations":2} 2022-03-07 14:53:41,745 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646664521745","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:35026","multi.service_calls":0,"queuetimems":5578,"class":"HRegionServer","multi.mutations":1} 2022-03-07 14:53:51,371 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646664231370","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 5 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:34978","multi.service_calls":0,"queuetimems":15204,"class":"HRegionServer","multi.mutations":5} 2022-03-07 14:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=346102, hits=337992, hitRatio=97.66%, , cachingAccesses=338122, cachingHits=337683, cachingHitsRatio=99.87%, evictions=21316, evicted=0, evictedPerRun=0.0 2022-03-07 14:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 14:57:18,886 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646664738886","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:36362","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-07 14:57:19,672 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646663839670","responsesize":"9926","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 6 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200002,"client":"172.30.41.155:36270","multi.service_calls":0,"queuetimems":800,"class":"HRegionServer","multi.mutations":6} 2022-03-07 14:57:25,400 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646664745398","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.155:36362","multi.service_calls":0,"queuetimems":6513,"class":"HRegionServer","multi.mutations":3} 2022-03-07 14:57:48,858 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646664768858","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:36362","multi.service_calls":0,"queuetimems":29972,"class":"HRegionServer","multi.mutations":1} 2022-03-07 14:58:03,342 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646664783342","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:36362","multi.service_calls":0,"queuetimems":44457,"class":"HRegionServer","multi.mutations":1} 2022-03-07 14:58:24,077 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646664204075","responsesize":"16531","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 10 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900002,"client":"172.30.41.118:34970","multi.service_calls":0,"queuetimems":65203,"class":"HRegionServer","multi.mutations":10} 2022-03-07 14:58:36,168 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646664516167","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:36336","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 14:58:41,746 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646664821745","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:36370","multi.service_calls":0,"queuetimems":5576,"class":"HRegionServer","multi.mutations":2} 2022-03-07 15:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=346904, hits=338794, hitRatio=97.66%, , cachingAccesses=338924, cachingHits=338485, cachingHitsRatio=99.87%, evictions=21346, evicted=0, evictedPerRun=0.0 2022-03-07 15:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 15:02:18,888 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646665038887","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:35116","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 15:02:25,401 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646665045400","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:36402","multi.service_calls":0,"queuetimems":6513,"class":"HRegionServer","multi.mutations":2} 2022-03-07 15:02:39,693 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646664759692","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:36362","multi.service_calls":0,"queuetimems":20807,"class":"HRegionServer","multi.mutations":2} 2022-03-07 15:03:03,344 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646665083343","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:35122","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 15:03:09,868 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646664789867","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:55598","multi.service_calls":0,"queuetimems":50981,"class":"HRegionServer","multi.mutations":2} 2022-03-07 15:03:11,277 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646664791276","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:35068","multi.service_calls":0,"queuetimems":52390,"class":"HRegionServer","multi.mutations":4} 2022-03-07 15:03:36,169 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646664816168","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":600001,"client":"172.30.41.118:35076","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 15:03:39,934 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646664219932","responsesize":"21490","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 13 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.31.0.219:55564","multi.service_calls":0,"queuetimems":3766,"class":"HRegionServer","multi.mutations":13} 2022-03-07 15:03:39,994 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646664519993","responsesize":"16531","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 10 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.31.0.219:55582","multi.service_calls":0,"queuetimems":3826,"class":"HRegionServer","multi.mutations":10} 2022-03-07 15:03:41,746 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646665121746","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:36404","multi.service_calls":0,"queuetimems":38403,"class":"HRegionServer","multi.mutations":3} 2022-03-07 15:03:51,372 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646664831371","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 5 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:55600","multi.service_calls":0,"queuetimems":15202,"class":"HRegionServer","multi.mutations":5} 2022-03-07 15:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=347724, hits=339614, hitRatio=97.67%, , cachingAccesses=339744, cachingHits=339305, cachingHitsRatio=99.87%, evictions=21376, evicted=0, evictedPerRun=0.0 2022-03-07 15:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 15:07:25,401 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646665345401","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:55632","multi.service_calls":0,"queuetimems":6051,"class":"HRegionServer","multi.mutations":1} 2022-03-07 15:07:39,693 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646665359693","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.31.0.219:55566","multi.service_calls":0,"queuetimems":27501,"class":"HRegionServer","multi.mutations":2} 2022-03-07 15:08:03,344 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646665383344","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:55632","multi.service_calls":0,"queuetimems":41931,"class":"HRegionServer","multi.mutations":1} 2022-03-07 15:08:09,868 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646665389868","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:35172","multi.service_calls":0,"queuetimems":47088,"class":"HRegionServer","multi.mutations":1} 2022-03-07 15:08:11,277 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646665391277","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:35172","multi.service_calls":0,"queuetimems":46005,"class":"HRegionServer","multi.mutations":1} 2022-03-07 15:08:24,079 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646665104078","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:36404","multi.service_calls":0,"queuetimems":20735,"class":"HRegionServer","multi.mutations":3} 2022-03-07 15:08:38,140 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646664818139","responsesize":"18183","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 11 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.30.41.118:35076","multi.service_calls":0,"queuetimems":1971,"class":"HRegionServer","multi.mutations":11} 2022-03-07 15:08:39,935 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646665419935","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:36446","multi.service_calls":0,"queuetimems":3765,"class":"HRegionServer","multi.mutations":1} 2022-03-07 15:08:41,747 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646665421747","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:35180","multi.service_calls":0,"queuetimems":5577,"class":"HRegionServer","multi.mutations":2} 2022-03-07 15:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=348517, hits=340407, hitRatio=97.67%, , cachingAccesses=340537, cachingHits=340098, cachingHitsRatio=99.87%, evictions=21406, evicted=0, evictedPerRun=0.0 2022-03-07 15:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 15:12:18,890 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646665338889","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:36440","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 15:12:19,674 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646665039672","responsesize":"19835","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 12 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900002,"client":"172.30.41.118:35116","multi.service_calls":0,"queuetimems":785,"class":"HRegionServer","multi.mutations":12} 2022-03-07 15:12:39,695 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646665659694","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:35224","multi.service_calls":0,"queuetimems":14292,"class":"HRegionServer","multi.mutations":1} 2022-03-07 15:12:48,860 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646665068858","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 8 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:55614","multi.service_calls":0,"queuetimems":29970,"class":"HRegionServer","multi.mutations":8} 2022-03-07 15:13:03,345 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646665683344","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:55666","multi.service_calls":0,"queuetimems":37941,"class":"HRegionServer","multi.mutations":3} 2022-03-07 15:13:36,170 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646665116168","responsesize":"16531","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 10 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.31.0.219:55616","multi.service_calls":0,"queuetimems":32824,"class":"HRegionServer","multi.mutations":10} 2022-03-07 15:13:36,171 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646665416170","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:36446","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 15:13:38,142 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646665718142","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:35234","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":2} 2022-03-07 15:13:39,936 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646665719935","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:36474","multi.service_calls":0,"queuetimems":1793,"class":"HRegionServer","multi.mutations":1} 2022-03-07 15:13:51,373 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646665431372","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 4 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:35180","multi.service_calls":0,"queuetimems":15202,"class":"HRegionServer","multi.mutations":4} 2022-03-07 15:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=349213, hits=341103, hitRatio=97.68%, , cachingAccesses=341233, cachingHits=340794, cachingHitsRatio=99.87%, evictions=21436, evicted=0, evictedPerRun=0.0 2022-03-07 15:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 15:17:25,404 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646665645402","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.155:36470","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 15:18:09,869 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646665689868","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:36470","multi.service_calls":0,"queuetimems":30174,"class":"HRegionServer","multi.mutations":2} 2022-03-07 15:18:36,170 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646666016170","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:35272","multi.service_calls":0,"queuetimems":76496,"class":"HRegionServer","multi.mutations":1} 2022-03-07 15:18:36,171 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646666016171","responsesize":"9917","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 6 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:55702","multi.service_calls":0,"queuetimems":77280,"class":"HRegionServer","multi.mutations":6} 2022-03-07 15:18:39,996 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646665419994","responsesize":"18183","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 11 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900002,"client":"172.31.0.219:55636","multi.service_calls":0,"queuetimems":3824,"class":"HRegionServer","multi.mutations":11} 2022-03-07 15:18:41,749 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646665721748","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:35234","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 15:18:51,374 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646666031374","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:36494","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 15:21:44,382 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=349982, hits=341872, hitRatio=97.68%, , cachingAccesses=342002, cachingHits=341563, cachingHitsRatio=99.87%, evictions=21466, evicted=0, evictedPerRun=0.0 2022-03-07 15:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 15:22:18,892 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646665938891","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:35272","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 15:22:19,675 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646665939674","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:36490","multi.service_calls":0,"queuetimems":783,"class":"HRegionServer","multi.mutations":2} 2022-03-07 15:22:25,406 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646666245405","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:35326","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 15:22:48,861 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646665968860","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:36490","multi.service_calls":0,"queuetimems":29969,"class":"HRegionServer","multi.mutations":4} 2022-03-07 15:23:03,346 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646665983345","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:55702","multi.service_calls":0,"queuetimems":44454,"class":"HRegionServer","multi.mutations":3} 2022-03-07 15:23:11,279 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646665691277","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 7 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.155:36470","multi.service_calls":0,"queuetimems":45875,"class":"HRegionServer","multi.mutations":7} 2022-03-07 15:23:24,082 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646665704080","responsesize":"21487","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 13 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900002,"client":"172.30.41.118:35224","multi.service_calls":0,"queuetimems":58678,"class":"HRegionServer","multi.mutations":13} 2022-03-07 15:23:38,143 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646666018142","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 9 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:35272","multi.service_calls":0,"queuetimems":78467,"class":"HRegionServer","multi.mutations":9} 2022-03-07 15:23:51,376 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646666331375","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:35334","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 15:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=350698, hits=342588, hitRatio=97.69%, , cachingAccesses=342718, cachingHits=342279, cachingHitsRatio=99.87%, evictions=21496, evicted=0, evictedPerRun=0.0 2022-03-07 15:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 15:27:19,676 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646666539676","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:35372","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 15:27:25,407 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646666545406","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.30.41.155:36558","multi.service_calls":0,"queuetimems":6483,"class":"HRegionServer","multi.mutations":2} 2022-03-07 15:27:39,696 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646665959695","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 5 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:35272","multi.service_calls":0,"queuetimems":20804,"class":"HRegionServer","multi.mutations":5} 2022-03-07 15:28:03,347 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646666583346","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:35376","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 15:28:11,280 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646666591280","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:35376","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 15:28:24,083 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646666604082","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:36560","multi.service_calls":0,"queuetimems":9549,"class":"HRegionServer","multi.mutations":1} 2022-03-07 15:28:36,171 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646666316170","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:55724","multi.service_calls":0,"queuetimems":70764,"class":"HRegionServer","multi.mutations":2} 2022-03-07 15:28:36,172 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646666316171","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:36522","multi.service_calls":0,"queuetimems":70766,"class":"HRegionServer","multi.mutations":2} 2022-03-07 15:28:38,144 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646666618143","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300000,"client":"172.30.41.118:35376","multi.service_calls":0,"queuetimems":26863,"class":"HRegionServer","multi.mutations":1} 2022-03-07 15:28:39,937 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646666019936","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 5 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:55702","multi.service_calls":0,"queuetimems":80261,"class":"HRegionServer","multi.mutations":5} 2022-03-07 15:28:39,997 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646666319996","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 9 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:55724","multi.service_calls":0,"queuetimems":74590,"class":"HRegionServer","multi.mutations":9} 2022-03-07 15:28:51,376 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646666631376","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:55744","multi.service_calls":0,"queuetimems":29693,"class":"HRegionServer","multi.mutations":1} 2022-03-07 15:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=351333, hits=343223, hitRatio=97.69%, , cachingAccesses=343353, cachingHits=342914, cachingHitsRatio=99.87%, evictions=21526, evicted=0, evictedPerRun=0.0 2022-03-07 15:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-07 15:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 15:32:18,894 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646666538893","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 6 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:35372","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":6} 2022-03-07 15:32:19,678 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646666839677","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:35418","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 15:32:39,696 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646666859696","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:36588","multi.service_calls":0,"queuetimems":20019,"class":"HRegionServer","multi.mutations":1} 2022-03-07 15:33:09,871 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646666289870","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 8 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":900001,"client":"172.30.41.118:35326","multi.service_calls":0,"queuetimems":44465,"class":"HRegionServer","multi.mutations":8} 2022-03-07 15:33:24,084 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646666904083","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:55756","multi.service_calls":0,"queuetimems":12802,"class":"HRegionServer","multi.mutations":1} 2022-03-07 15:33:36,172 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646666916172","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:35440","multi.service_calls":0,"queuetimems":24861,"class":"HRegionServer","multi.mutations":2} 2022-03-07 15:33:41,750 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646666321749","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:36522","multi.service_calls":0,"queuetimems":76343,"class":"HRegionServer","multi.mutations":7} 2022-03-07 15:33:51,377 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646666931377","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300000,"client":"172.30.41.118:35438","multi.service_calls":0,"queuetimems":15168,"class":"HRegionServer","multi.mutations":1} 2022-03-07 15:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=352053, hits=343943, hitRatio=97.70%, , cachingAccesses=344073, cachingHits=343634, cachingHitsRatio=99.87%, evictions=21556, evicted=0, evictedPerRun=0.0 2022-03-07 15:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 15:37:19,679 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646667139678","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:36604","multi.service_calls":0,"queuetimems":783,"class":"HRegionServer","multi.mutations":1} 2022-03-07 15:37:25,409 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646666845408","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 4 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:35418","multi.service_calls":0,"queuetimems":5731,"class":"HRegionServer","multi.mutations":4} 2022-03-07 15:37:48,862 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646666568861","responsesize":"19835","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 12 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.30.41.118:35372","multi.service_calls":0,"queuetimems":29185,"class":"HRegionServer","multi.mutations":12} 2022-03-07 15:38:03,348 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646666883347","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:36588","multi.service_calls":0,"queuetimems":43669,"class":"HRegionServer","multi.mutations":5} 2022-03-07 15:38:11,282 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646666891281","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:55756","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 15:38:36,172 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646666916172","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:35444","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 15:38:36,173 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646667216173","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300000,"client":"172.30.41.118:35504","multi.service_calls":0,"queuetimems":12088,"class":"HRegionServer","multi.mutations":2} 2022-03-07 15:38:39,938 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646666919937","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:36596","multi.service_calls":0,"queuetimems":3765,"class":"HRegionServer","multi.mutations":3} 2022-03-07 15:38:39,998 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646666919997","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:55756","multi.service_calls":0,"queuetimems":3825,"class":"HRegionServer","multi.mutations":2} 2022-03-07 15:38:41,750 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646667221750","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:36608","multi.service_calls":0,"queuetimems":17665,"class":"HRegionServer","multi.mutations":1} 2022-03-07 15:38:51,378 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646667231377","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:36608","multi.service_calls":0,"queuetimems":27292,"class":"HRegionServer","multi.mutations":1} 2022-03-07 15:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=352916, hits=344806, hitRatio=97.70%, , cachingAccesses=344936, cachingHits=344497, cachingHitsRatio=99.87%, evictions=21586, evicted=0, evictedPerRun=0.0 2022-03-07 15:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 15:42:19,680 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646667439680","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:35568","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":2} 2022-03-07 15:43:09,873 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646667189872","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:35504","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 15:43:11,283 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646667491282","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:36638","multi.service_calls":0,"queuetimems":7933,"class":"HRegionServer","multi.mutations":1} 2022-03-07 15:43:36,174 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646667516173","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:36638","multi.service_calls":0,"queuetimems":32824,"class":"HRegionServer","multi.mutations":2} 2022-03-07 15:43:39,998 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646667519998","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:36638","multi.service_calls":0,"queuetimems":28715,"class":"HRegionServer","multi.mutations":1} 2022-03-07 15:43:51,378 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646667531378","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:55788","multi.service_calls":0,"queuetimems":40095,"class":"HRegionServer","multi.mutations":1} 2022-03-07 15:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=353664, hits=345554, hitRatio=97.71%, , cachingAccesses=345684, cachingHits=345245, cachingHitsRatio=99.87%, evictions=21616, evicted=0, evictedPerRun=0.0 2022-03-07 15:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 15:47:18,897 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646667138895","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:35488","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":5} 2022-03-07 15:47:25,411 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646667445409","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.155:36632","multi.service_calls":0,"queuetimems":5729,"class":"HRegionServer","multi.mutations":2} 2022-03-07 15:47:39,698 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646667159697","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:55770","multi.service_calls":0,"queuetimems":20801,"class":"HRegionServer","multi.mutations":3} 2022-03-07 15:47:48,863 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646667468862","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 9 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:55786","multi.service_calls":0,"queuetimems":29182,"class":"HRegionServer","multi.mutations":9} 2022-03-07 15:48:09,874 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646667789874","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:35636","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 15:48:11,284 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646667791283","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:55806","multi.service_calls":0,"queuetimems":1409,"class":"HRegionServer","multi.mutations":2} 2022-03-07 15:48:36,205 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646667816205","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.155:36674","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 15:48:39,939 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646667519938","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:55788","multi.service_calls":0,"queuetimems":36589,"class":"HRegionServer","multi.mutations":3} 2022-03-07 15:48:41,752 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646667521751","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 8 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.31.0.219:55788","multi.service_calls":0,"queuetimems":38401,"class":"HRegionServer","multi.mutations":8} 2022-03-07 15:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=354441, hits=346331, hitRatio=97.71%, , cachingAccesses=346461, cachingHits=346022, cachingHitsRatio=99.87%, evictions=21646, evicted=0, evictedPerRun=0.0 2022-03-07 15:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 15:52:18,898 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646668038897","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:35696","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 15:52:39,699 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646668059698","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:35696","multi.service_calls":0,"queuetimems":20800,"class":"HRegionServer","multi.mutations":1} 2022-03-07 15:53:03,350 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646667483349","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 5 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:35576","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":5} 2022-03-07 15:53:11,284 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646668091284","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:55820","multi.service_calls":0,"queuetimems":31585,"class":"HRegionServer","multi.mutations":1} 2022-03-07 15:53:24,088 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646667204085","responsesize":"21490","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 13 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.118:35504","multi.service_calls":0,"queuetimems":14213,"class":"HRegionServer","multi.mutations":13} 2022-03-07 15:53:36,174 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646667516173","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 8 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:35576","multi.service_calls":0,"queuetimems":32824,"class":"HRegionServer","multi.mutations":8} 2022-03-07 15:53:38,146 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646666918144","responsesize":"28101","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 17 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1500001,"client":"172.30.41.118:35444","multi.service_calls":0,"queuetimems":1972,"class":"HRegionServer","multi.mutations":17} 2022-03-07 15:53:39,999 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646667819998","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:36672","multi.service_calls":0,"queuetimems":30124,"class":"HRegionServer","multi.mutations":4} 2022-03-07 15:53:41,752 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646668121752","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:36692","multi.service_calls":0,"queuetimems":5546,"class":"HRegionServer","multi.mutations":1} 2022-03-07 15:53:51,381 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646667831380","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.31.0.219:55808","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":6} 2022-03-07 15:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=355170, hits=347060, hitRatio=97.72%, , cachingAccesses=347190, cachingHits=346751, cachingHitsRatio=99.87%, evictions=21676, evicted=0, evictedPerRun=0.0 2022-03-07 15:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 15:57:19,682 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646667739681","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:35620","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":6} 2022-03-07 15:57:25,413 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646668045411","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.155:36690","multi.service_calls":0,"queuetimems":6513,"class":"HRegionServer","multi.mutations":4} 2022-03-07 15:57:48,864 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646668068863","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:35696","multi.service_calls":0,"queuetimems":9164,"class":"HRegionServer","multi.mutations":2} 2022-03-07 15:58:09,875 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646668089874","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:36690","multi.service_calls":0,"queuetimems":50976,"class":"HRegionServer","multi.mutations":2} 2022-03-07 15:58:24,088 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646668404088","responsesize":"6613","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:36708","multi.service_calls":0,"queuetimems":65189,"class":"HRegionServer","multi.mutations":4} 2022-03-07 15:58:36,175 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646668416175","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:36714","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 15:58:39,940 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646668119939","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:35708","multi.service_calls":0,"queuetimems":3733,"class":"HRegionServer","multi.mutations":6} 2022-03-07 15:58:41,753 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646668421753","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:35786","multi.service_calls":0,"queuetimems":5578,"class":"HRegionServer","multi.mutations":1} 2022-03-07 15:58:41,753 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646668421753","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300000,"client":"172.30.41.118:35786","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 15:58:51,381 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646668431381","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:35786","multi.service_calls":0,"queuetimems":15206,"class":"HRegionServer","multi.mutations":2} 2022-03-07 16:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=355931, hits=347821, hitRatio=97.72%, , cachingAccesses=347951, cachingHits=347512, cachingHitsRatio=99.87%, evictions=21706, evicted=0, evictedPerRun=0.0 2022-03-07 16:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 16:02:18,900 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646668338899","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 6 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:35760","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":6} 2022-03-07 16:02:19,684 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646668639683","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:35840","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 16:02:39,700 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646668359699","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:36708","multi.service_calls":0,"queuetimems":20800,"class":"HRegionServer","multi.mutations":3} 2022-03-07 16:03:24,090 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646668704088","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.118:35840","multi.service_calls":0,"queuetimems":58674,"class":"HRegionServer","multi.mutations":1} 2022-03-07 16:03:36,178 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646668716177","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300001,"client":"172.30.41.118:35854","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 16:03:36,207 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646668116206","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 8 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:35708","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":8} 2022-03-07 16:03:41,754 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646668721754","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:35854","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 16:03:51,382 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646668731381","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:35854","multi.service_calls":0,"queuetimems":15204,"class":"HRegionServer","multi.mutations":2} 2022-03-07 16:04:49,270 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2257ms No GCs detected 2022-03-07 16:04:52,322 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2052ms No GCs detected 2022-03-07 16:04:55,501 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1438ms No GCs detected 2022-03-07 16:05:02,694 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-07 16:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=356659, hits=348549, hitRatio=97.73%, , cachingAccesses=348679, cachingHits=348240, cachingHitsRatio=99.87%, evictions=21736, evicted=0, evictedPerRun=0.0 2022-03-07 16:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 16:07:18,903 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646668938902","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:36746","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 16:07:25,415 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646668645414","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:35840","multi.service_calls":0,"queuetimems":5731,"class":"HRegionServer","multi.mutations":4} 2022-03-07 16:08:03,352 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646668383350","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.31.0.219:55834","multi.service_calls":0,"queuetimems":44450,"class":"HRegionServer","multi.mutations":6} 2022-03-07 16:08:11,286 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646668391284","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 7 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.118:35760","multi.service_calls":0,"queuetimems":52385,"class":"HRegionServer","multi.mutations":7} 2022-03-07 16:08:24,092 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646669004091","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:36750","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 16:08:36,179 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646669016178","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:35908","multi.service_calls":0,"queuetimems":12086,"class":"HRegionServer","multi.mutations":1} 2022-03-07 16:08:41,755 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646668721754","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:36734","multi.service_calls":0,"queuetimems":5577,"class":"HRegionServer","multi.mutations":3} 2022-03-07 16:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=357449, hits=349339, hitRatio=97.73%, , cachingAccesses=349469, cachingHits=349030, cachingHitsRatio=99.87%, evictions=21766, evicted=0, evictedPerRun=0.0 2022-03-07 16:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 16:12:18,904 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646669238904","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:35976","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 16:12:19,686 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646668939684","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 5 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:35892","multi.service_calls":0,"queuetimems":782,"class":"HRegionServer","multi.mutations":5} 2022-03-07 16:12:39,701 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646668959700","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:35892","multi.service_calls":0,"queuetimems":20798,"class":"HRegionServer","multi.mutations":3} 2022-03-07 16:12:48,866 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646668668864","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:55852","multi.service_calls":0,"queuetimems":29180,"class":"HRegionServer","multi.mutations":3} 2022-03-07 16:13:09,877 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646668689875","responsesize":"19835","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 12 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900002,"client":"172.30.41.118:35840","multi.service_calls":0,"queuetimems":44461,"class":"HRegionServer","multi.mutations":12} 2022-03-07 16:13:36,209 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646669016208","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:36750","multi.service_calls":0,"queuetimems":12116,"class":"HRegionServer","multi.mutations":2} 2022-03-07 16:13:38,148 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646668418146","responsesize":"21490","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 13 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200001,"client":"172.31.0.219:55838","multi.service_calls":0,"queuetimems":1971,"class":"HRegionServer","multi.mutations":13} 2022-03-07 16:13:41,755 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646669321755","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:35982","multi.service_calls":0,"queuetimems":30468,"class":"HRegionServer","multi.mutations":1} 2022-03-07 16:13:41,755 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646669021754","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 5 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:36750","multi.service_calls":0,"queuetimems":17662,"class":"HRegionServer","multi.mutations":5} 2022-03-07 16:13:51,383 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646669031382","responsesize":"16528","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 10 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":600001,"client":"172.30.41.118:35908","multi.service_calls":0,"queuetimems":27290,"class":"HRegionServer","multi.mutations":10} 2022-03-07 16:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=358248, hits=350138, hitRatio=97.74%, , cachingAccesses=350268, cachingHits=349829, cachingHitsRatio=99.87%, evictions=21796, evicted=0, evictedPerRun=0.0 2022-03-07 16:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 16:17:18,906 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646669538905","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:36038","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 16:17:19,687 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646669539686","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:36784","multi.service_calls":0,"queuetimems":781,"class":"HRegionServer","multi.mutations":1} 2022-03-07 16:17:25,417 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646669245415","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 6 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.155:36766","multi.service_calls":0,"queuetimems":6511,"class":"HRegionServer","multi.mutations":6} 2022-03-07 16:17:39,702 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646669559701","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:36784","multi.service_calls":0,"queuetimems":20796,"class":"HRegionServer","multi.mutations":3} 2022-03-07 16:17:48,867 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646669568867","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:36784","multi.service_calls":0,"queuetimems":29180,"class":"HRegionServer","multi.mutations":1} 2022-03-07 16:18:03,354 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646669283353","responsesize":"11572","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 7 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:35982","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":7} 2022-03-07 16:18:36,180 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646669316179","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:36768","multi.service_calls":0,"queuetimems":32826,"class":"HRegionServer","multi.mutations":3} 2022-03-07 16:18:38,148 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646669618148","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:36056","multi.service_calls":0,"queuetimems":1939,"class":"HRegionServer","multi.mutations":1} 2022-03-07 16:18:41,756 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646668721754","responsesize":"23142","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 14 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200001,"client":"172.30.41.118:35854","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":14} 2022-03-07 16:18:41,756 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646669621755","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:36056","multi.service_calls":0,"queuetimems":5545,"class":"HRegionServer","multi.mutations":3} 2022-03-07 16:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=359014, hits=350904, hitRatio=97.74%, , cachingAccesses=351034, cachingHits=350595, cachingHitsRatio=99.87%, evictions=21826, evicted=0, evictedPerRun=0.0 2022-03-07 16:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 16:22:18,907 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646669838907","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:36806","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 16:23:03,355 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646669883354","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:36130","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 16:23:09,879 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646669589878","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:36056","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 16:23:36,210 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646669616209","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 5 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":600001,"client":"172.30.41.118:36056","multi.service_calls":0,"queuetimems":26331,"class":"HRegionServer","multi.mutations":5} 2022-03-07 16:23:38,150 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646669918149","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:36134","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 16:23:41,756 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646669621755","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 4 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:55908","multi.service_calls":0,"queuetimems":31877,"class":"HRegionServer","multi.mutations":4} 2022-03-07 16:23:41,756 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646669921756","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:36138","multi.service_calls":0,"queuetimems":3577,"class":"HRegionServer","multi.mutations":2} 2022-03-07 16:23:41,785 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646669921785","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300000,"client":"172.30.41.118:36136","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 16:23:51,384 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646669631383","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 8 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:36056","multi.service_calls":0,"queuetimems":15173,"class":"HRegionServer","multi.mutations":8} 2022-03-07 16:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=359786, hits=351676, hitRatio=97.75%, , cachingAccesses=351806, cachingHits=351367, cachingHitsRatio=99.88%, evictions=21856, evicted=0, evictedPerRun=0.0 2022-03-07 16:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 16:27:18,909 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646670138908","responsesize":"6613","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:36184","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-07 16:27:19,688 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646669839687","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 9 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:36124","multi.service_calls":0,"queuetimems":780,"class":"HRegionServer","multi.mutations":9} 2022-03-07 16:27:25,419 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646669845417","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.155:36806","multi.service_calls":0,"queuetimems":6510,"class":"HRegionServer","multi.mutations":4} 2022-03-07 16:27:48,868 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646669868867","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:55924","multi.service_calls":0,"queuetimems":29960,"class":"HRegionServer","multi.mutations":2} 2022-03-07 16:28:03,356 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646670183356","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:36826","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 16:28:11,294 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646669291287","responsesize":"13230","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 8 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200007,"client":"172.30.41.118:35982","multi.service_calls":0,"queuetimems":7934,"class":"HRegionServer","multi.mutations":8} 2022-03-07 16:28:24,095 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646669304092","responsesize":"11578","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 7 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200003,"client":"172.31.0.219:55888","multi.service_calls":0,"queuetimems":20739,"class":"HRegionServer","multi.mutations":7} 2022-03-07 16:28:38,150 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646670218150","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:36194","multi.service_calls":0,"queuetimems":34794,"class":"HRegionServer","multi.mutations":1} 2022-03-07 16:28:41,788 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646670221787","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:36200","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 16:28:41,807 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646670221806","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300001,"client":"172.30.41.118:36202","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 16:28:51,385 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646670231385","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:55940","multi.service_calls":0,"queuetimems":48029,"class":"HRegionServer","multi.mutations":2} 2022-03-07 16:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=360589, hits=352479, hitRatio=97.75%, , cachingAccesses=352609, cachingHits=352170, cachingHitsRatio=99.88%, evictions=21886, evicted=0, evictedPerRun=0.0 2022-03-07 16:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-07 16:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 16:32:39,704 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646669859702","responsesize":"19835","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 12 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.31.0.219:55924","multi.service_calls":0,"queuetimems":20795,"class":"HRegionServer","multi.mutations":12} 2022-03-07 16:32:48,868 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646670468868","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:36248","multi.service_calls":0,"queuetimems":29958,"class":"HRegionServer","multi.mutations":1} 2022-03-07 16:33:03,358 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646670483357","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:36850","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 16:33:09,880 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646670189879","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 8 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:36194","multi.service_calls":0,"queuetimems":6523,"class":"HRegionServer","multi.mutations":8} 2022-03-07 16:33:11,295 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646670491295","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:36260","multi.service_calls":0,"queuetimems":7938,"class":"HRegionServer","multi.mutations":2} 2022-03-07 16:33:24,097 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646670504095","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:36850","multi.service_calls":0,"queuetimems":20738,"class":"HRegionServer","multi.mutations":1} 2022-03-07 16:33:36,182 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646669916180","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 5 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:55926","multi.service_calls":0,"queuetimems":32825,"class":"HRegionServer","multi.mutations":5} 2022-03-07 16:33:41,788 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646670521788","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:36850","multi.service_calls":0,"queuetimems":38431,"class":"HRegionServer","multi.mutations":1} 2022-03-07 16:33:41,807 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646670521807","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:36850","multi.service_calls":0,"queuetimems":38450,"class":"HRegionServer","multi.mutations":1} 2022-03-07 16:33:51,386 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646670531385","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:36850","multi.service_calls":0,"queuetimems":48028,"class":"HRegionServer","multi.mutations":1} 2022-03-07 16:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=361342, hits=353232, hitRatio=97.76%, , cachingAccesses=353362, cachingHits=352923, cachingHitsRatio=99.88%, evictions=21916, evicted=0, evictedPerRun=0.0 2022-03-07 16:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 16:37:18,911 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646670438910","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:36842","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 16:37:19,690 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646670439689","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 9 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:36248","multi.service_calls":0,"queuetimems":779,"class":"HRegionServer","multi.mutations":9} 2022-03-07 16:37:25,420 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646670445419","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:36248","multi.service_calls":0,"queuetimems":6509,"class":"HRegionServer","multi.mutations":3} 2022-03-07 16:37:39,705 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646670759705","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:36310","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 16:38:03,358 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646670783358","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:36886","multi.service_calls":0,"queuetimems":23653,"class":"HRegionServer","multi.mutations":3} 2022-03-07 16:38:09,880 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646670789880","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:55976","multi.service_calls":0,"queuetimems":30175,"class":"HRegionServer","multi.mutations":1} 2022-03-07 16:38:11,296 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646670791295","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:55976","multi.service_calls":0,"queuetimems":31590,"class":"HRegionServer","multi.mutations":1} 2022-03-07 16:38:36,212 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646670216210","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.31.0.219:55940","multi.service_calls":0,"queuetimems":77301,"class":"HRegionServer","multi.mutations":4} 2022-03-07 16:38:38,151 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646670518150","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 5 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":600001,"client":"172.30.41.118:36260","multi.service_calls":0,"queuetimems":34793,"class":"HRegionServer","multi.mutations":5} 2022-03-07 16:38:41,757 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646670221756","responsesize":"18183","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 11 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.31.0.219:55940","multi.service_calls":0,"queuetimems":82847,"class":"HRegionServer","multi.mutations":11} 2022-03-07 16:38:41,789 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646670821788","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:36316","multi.service_calls":0,"queuetimems":5605,"class":"HRegionServer","multi.mutations":2} 2022-03-07 16:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=362094, hits=353984, hitRatio=97.76%, , cachingAccesses=354114, cachingHits=353675, cachingHitsRatio=99.88%, evictions=21946, evicted=0, evictedPerRun=0.0 2022-03-07 16:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 16:42:18,915 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646671038912","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300003,"client":"172.30.41.118:36352","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 16:42:19,691 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646671039691","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:36352","multi.service_calls":0,"queuetimems":779,"class":"HRegionServer","multi.mutations":2} 2022-03-07 16:42:25,422 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646671045420","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.155:36920","multi.service_calls":0,"queuetimems":6508,"class":"HRegionServer","multi.mutations":1} 2022-03-07 16:42:48,870 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646670768869","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 6 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:36310","multi.service_calls":0,"queuetimems":9164,"class":"HRegionServer","multi.mutations":6} 2022-03-07 16:43:09,881 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646671089880","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:55990","multi.service_calls":0,"queuetimems":50189,"class":"HRegionServer","multi.mutations":1} 2022-03-07 16:43:36,183 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646670816183","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:36894","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 16:43:41,758 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646671121758","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:36930","multi.service_calls":0,"queuetimems":5545,"class":"HRegionServer","multi.mutations":2} 2022-03-07 16:43:51,387 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646670831386","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 8 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:36316","multi.service_calls":0,"queuetimems":15203,"class":"HRegionServer","multi.mutations":8} 2022-03-07 16:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=362852, hits=354742, hitRatio=97.76%, , cachingAccesses=354872, cachingHits=354433, cachingHitsRatio=99.88%, evictions=21976, evicted=0, evictedPerRun=0.0 2022-03-07 16:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 16:47:25,422 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646671345422","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:56006","multi.service_calls":0,"queuetimems":6505,"class":"HRegionServer","multi.mutations":1} 2022-03-07 16:47:39,706 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646671059705","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:36920","multi.service_calls":0,"queuetimems":20793,"class":"HRegionServer","multi.mutations":5} 2022-03-07 16:47:48,870 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646671368870","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:56006","multi.service_calls":0,"queuetimems":29953,"class":"HRegionServer","multi.mutations":1} 2022-03-07 16:48:09,881 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646671389881","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:56006","multi.service_calls":0,"queuetimems":50964,"class":"HRegionServer","multi.mutations":1} 2022-03-07 16:48:24,099 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646670804097","responsesize":"18183","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 11 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900002,"client":"172.30.41.118:36310","multi.service_calls":0,"queuetimems":35228,"class":"HRegionServer","multi.mutations":11} 2022-03-07 16:48:36,213 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646671116213","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:36362","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 16:48:38,153 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646671118152","responsesize":"11572","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 7 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:36362","multi.service_calls":0,"queuetimems":1939,"class":"HRegionServer","multi.mutations":7} 2022-03-07 16:48:41,759 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646671421759","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:36414","multi.service_calls":0,"queuetimems":5575,"class":"HRegionServer","multi.mutations":1} 2022-03-07 16:48:41,789 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646671121789","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 5 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.31.0.219:55994","multi.service_calls":0,"queuetimems":5576,"class":"HRegionServer","multi.mutations":5} 2022-03-07 16:48:51,387 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646671431387","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:36964","multi.service_calls":0,"queuetimems":15203,"class":"HRegionServer","multi.mutations":1} 2022-03-07 16:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=363550, hits=355440, hitRatio=97.77%, , cachingAccesses=355570, cachingHits=355131, cachingHitsRatio=99.88%, evictions=22006, evicted=0, evictedPerRun=0.0 2022-03-07 16:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 16:52:19,693 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646671339691","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.118:36398","multi.service_calls":0,"queuetimems":774,"class":"HRegionServer","multi.mutations":3} 2022-03-07 16:52:25,424 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646671645423","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:36990","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 16:52:39,707 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646671659707","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:36256","multi.service_calls":0,"queuetimems":26358,"class":"HRegionServer","multi.mutations":2} 2022-03-07 16:52:48,871 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646671668871","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:36990","multi.service_calls":0,"queuetimems":22311,"class":"HRegionServer","multi.mutations":1} 2022-03-07 16:53:03,360 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646671083358","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 8 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:55990","multi.service_calls":0,"queuetimems":44445,"class":"HRegionServer","multi.mutations":8} 2022-03-07 16:53:11,297 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646671091296","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 6 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:55990","multi.service_calls":0,"queuetimems":52384,"class":"HRegionServer","multi.mutations":6} 2022-03-07 16:53:24,100 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646671704099","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:36990","multi.service_calls":0,"queuetimems":57539,"class":"HRegionServer","multi.mutations":1} 2022-03-07 16:53:36,214 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646671716214","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:36990","multi.service_calls":0,"queuetimems":66557,"class":"HRegionServer","multi.mutations":1} 2022-03-07 16:53:38,153 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646671718153","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:36990","multi.service_calls":0,"queuetimems":71287,"class":"HRegionServer","multi.mutations":1} 2022-03-07 16:53:41,809 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646670821807","responsesize":"13230","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 8 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200002,"client":"172.30.41.155:36894","multi.service_calls":0,"queuetimems":5624,"class":"HRegionServer","multi.mutations":8} 2022-03-07 16:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=364282, hits=356172, hitRatio=97.77%, , cachingAccesses=356302, cachingHits=355863, cachingHitsRatio=99.88%, evictions=22036, evicted=0, evictedPerRun=0.0 2022-03-07 16:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 16:57:18,918 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646671338917","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 6 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:36398","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":6} 2022-03-07 16:58:03,360 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646671983360","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:36518","multi.service_calls":0,"queuetimems":37935,"class":"HRegionServer","multi.mutations":1} 2022-03-07 16:58:11,298 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646671991297","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:37010","multi.service_calls":0,"queuetimems":45872,"class":"HRegionServer","multi.mutations":1} 2022-03-07 16:58:36,186 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646671416184","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 6 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.155:36964","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":6} 2022-03-07 16:58:38,154 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646672018154","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:56040","multi.service_calls":0,"queuetimems":1938,"class":"HRegionServer","multi.mutations":1} 2022-03-07 16:58:41,761 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646671721760","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 5 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:36468","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":5} 2022-03-07 16:58:41,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646671721790","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 5 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:36996","multi.service_calls":0,"queuetimems":30,"class":"HRegionServer","multi.mutations":5} 2022-03-07 16:58:41,809 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646672021809","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:56040","multi.service_calls":0,"queuetimems":5593,"class":"HRegionServer","multi.mutations":1} 2022-03-07 17:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=364993, hits=356883, hitRatio=97.78%, , cachingAccesses=357013, cachingHits=356574, cachingHitsRatio=99.88%, evictions=22066, evicted=0, evictedPerRun=0.0 2022-03-07 17:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 17:02:18,920 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646672238919","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:36588","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 17:02:19,695 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646671939694","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 4 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:37010","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-07 17:02:25,428 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646671945424","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 6 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600004,"client":"172.30.41.118:36518","multi.service_calls":0,"queuetimems":5729,"class":"HRegionServer","multi.mutations":6} 2022-03-07 17:03:09,882 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646671689881","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:56024","multi.service_calls":0,"queuetimems":44458,"class":"HRegionServer","multi.mutations":3} 2022-03-07 17:03:11,298 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646672291298","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:37030","multi.service_calls":0,"queuetimems":11521,"class":"HRegionServer","multi.mutations":1} 2022-03-07 17:03:36,216 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646672016215","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:37014","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 17:03:41,761 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646672321761","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:36602","multi.service_calls":0,"queuetimems":26928,"class":"HRegionServer","multi.mutations":1} 2022-03-07 17:03:41,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646672321791","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:56056","multi.service_calls":0,"queuetimems":30492,"class":"HRegionServer","multi.mutations":3} 2022-03-07 17:03:41,810 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646672321809","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:36602","multi.service_calls":0,"queuetimems":5623,"class":"HRegionServer","multi.mutations":1} 2022-03-07 17:03:51,389 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646671731388","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 8 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:36468","multi.service_calls":0,"queuetimems":9628,"class":"HRegionServer","multi.mutations":8} 2022-03-07 17:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=365770, hits=357660, hitRatio=97.78%, , cachingAccesses=357790, cachingHits=357351, cachingHitsRatio=99.88%, evictions=22096, evicted=0, evictedPerRun=0.0 2022-03-07 17:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 17:07:25,429 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646672545428","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:37050","multi.service_calls":0,"queuetimems":6507,"class":"HRegionServer","multi.mutations":1} 2022-03-07 17:07:39,709 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646671959707","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.31.0.219:56036","multi.service_calls":0,"queuetimems":20012,"class":"HRegionServer","multi.mutations":4} 2022-03-07 17:07:48,872 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646671968871","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 5 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:56036","multi.service_calls":0,"queuetimems":29176,"class":"HRegionServer","multi.mutations":5} 2022-03-07 17:08:03,361 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646672283360","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:37030","multi.service_calls":0,"queuetimems":44441,"class":"HRegionServer","multi.mutations":2} 2022-03-07 17:08:09,884 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646672589883","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:37054","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 17:08:11,299 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646672591299","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:36666","multi.service_calls":0,"queuetimems":1416,"class":"HRegionServer","multi.mutations":1} 2022-03-07 17:08:24,102 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646672004100","responsesize":"19835","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 12 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900002,"client":"172.30.41.118:36518","multi.service_calls":0,"queuetimems":58675,"class":"HRegionServer","multi.mutations":12} 2022-03-07 17:08:36,187 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646672316186","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":600001,"client":"172.30.41.118:36602","multi.service_calls":0,"queuetimems":24888,"class":"HRegionServer","multi.mutations":2} 2022-03-07 17:08:36,217 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646672616216","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:37054","multi.service_calls":0,"queuetimems":24917,"class":"HRegionServer","multi.mutations":1} 2022-03-07 17:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=366411, hits=358301, hitRatio=97.79%, , cachingAccesses=358431, cachingHits=357992, cachingHitsRatio=99.88%, evictions=22126, evicted=0, evictedPerRun=0.0 2022-03-07 17:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 17:12:18,922 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646672538921","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 4 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:36652","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-07 17:12:19,697 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646672539696","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 6 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:36652","multi.service_calls":0,"queuetimems":775,"class":"HRegionServer","multi.mutations":6} 2022-03-07 17:12:25,431 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646672845430","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:37070","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 17:13:36,218 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646672916217","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:37070","multi.service_calls":0,"queuetimems":56507,"class":"HRegionServer","multi.mutations":1} 2022-03-07 17:13:38,156 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646672318154","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 7 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:36602","multi.service_calls":0,"queuetimems":26856,"class":"HRegionServer","multi.mutations":7} 2022-03-07 17:13:41,793 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646672621792","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:37056","multi.service_calls":0,"queuetimems":30,"class":"HRegionServer","multi.mutations":3} 2022-03-07 17:13:41,811 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646672621810","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:56078","multi.service_calls":0,"queuetimems":48,"class":"HRegionServer","multi.mutations":4} 2022-03-07 17:13:51,390 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646672631389","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 4 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:37056","multi.service_calls":0,"queuetimems":9627,"class":"HRegionServer","multi.mutations":4} 2022-03-07 17:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=367239, hits=359129, hitRatio=97.79%, , cachingAccesses=359259, cachingHits=358820, cachingHitsRatio=99.88%, evictions=22156, evicted=0, evictedPerRun=0.0 2022-03-07 17:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 17:17:39,711 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646672859710","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":600000,"client":"172.30.41.118:36724","multi.service_calls":0,"queuetimems":14280,"class":"HRegionServer","multi.mutations":2} 2022-03-07 17:17:48,874 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646672868873","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:37070","multi.service_calls":0,"queuetimems":23443,"class":"HRegionServer","multi.mutations":4} 2022-03-07 17:18:09,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646672889884","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 4 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:37070","multi.service_calls":0,"queuetimems":30174,"class":"HRegionServer","multi.mutations":4} 2022-03-07 17:18:36,188 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646672916187","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 8 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:36724","multi.service_calls":0,"queuetimems":56477,"class":"HRegionServer","multi.mutations":8} 2022-03-07 17:18:41,763 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646672621762","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":900001,"client":"172.30.41.118:36676","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-07 17:18:41,811 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646673221811","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.31.0.219:56130","multi.service_calls":0,"queuetimems":5562,"class":"HRegionServer","multi.mutations":2} 2022-03-07 17:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=367899, hits=359789, hitRatio=97.80%, , cachingAccesses=359919, cachingHits=359480, cachingHitsRatio=99.88%, evictions=22186, evicted=0, evictedPerRun=0.0 2022-03-07 17:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 17:22:18,923 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646673138922","responsesize":"11572","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 7 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:36770","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":7} 2022-03-07 17:22:19,698 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646673139697","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:37084","multi.service_calls":0,"queuetimems":774,"class":"HRegionServer","multi.mutations":4} 2022-03-07 17:22:25,432 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646673145431","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:37084","multi.service_calls":0,"queuetimems":6509,"class":"HRegionServer","multi.mutations":2} 2022-03-07 17:22:39,712 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646673459712","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:37104","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 17:22:48,875 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646673468874","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:36846","multi.service_calls":0,"queuetimems":9162,"class":"HRegionServer","multi.mutations":1} 2022-03-07 17:23:03,363 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646672883361","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 6 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.31.0.219:56090","multi.service_calls":0,"queuetimems":37931,"class":"HRegionServer","multi.mutations":6} 2022-03-07 17:23:09,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646673489885","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300000,"client":"172.30.41.118:36846","multi.service_calls":0,"queuetimems":21010,"class":"HRegionServer","multi.mutations":1} 2022-03-07 17:23:11,301 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646672891299","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 8 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.118:36724","multi.service_calls":0,"queuetimems":45869,"class":"HRegionServer","multi.mutations":8} 2022-03-07 17:23:24,105 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646672904102","responsesize":"14879","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 9 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900003,"client":"172.31.0.219:56090","multi.service_calls":0,"queuetimems":58671,"class":"HRegionServer","multi.mutations":9} 2022-03-07 17:23:36,190 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646673516189","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:36850","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 17:23:36,220 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646673216219","responsesize":"18180","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 11 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":600000,"client":"172.30.41.118:36792","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":11} 2022-03-07 17:23:38,157 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646673218156","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:37090","multi.service_calls":0,"queuetimems":1937,"class":"HRegionServer","multi.mutations":3} 2022-03-07 17:23:41,765 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646673521764","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300000,"client":"172.30.41.118:36850","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 17:23:41,794 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646673221793","responsesize":"11572","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:56132","multi.service_calls":0,"queuetimems":5574,"class":"HRegionServer","multi.mutations":7} 2022-03-07 17:23:43,697 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646673523696","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.30.41.155:36992","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 17:23:51,391 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646673231390","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 6 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:56132","multi.service_calls":0,"queuetimems":15171,"class":"HRegionServer","multi.mutations":6} 2022-03-07 17:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=368711, hits=360601, hitRatio=97.80%, , cachingAccesses=360731, cachingHits=360292, cachingHitsRatio=99.88%, evictions=22216, evicted=0, evictedPerRun=0.0 2022-03-07 17:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 17:27:18,925 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646673738924","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:36882","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 17:27:19,700 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646673739699","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:36882","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 17:27:25,434 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646673745432","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.155:37130","multi.service_calls":0,"queuetimems":6508,"class":"HRegionServer","multi.mutations":1} 2022-03-07 17:27:48,875 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646673768875","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:37130","multi.service_calls":0,"queuetimems":29951,"class":"HRegionServer","multi.mutations":3} 2022-03-07 17:28:03,364 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646673783364","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:37136","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 17:28:09,916 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646673789915","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300000,"client":"172.30.41.118:36894","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 17:28:36,220 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646673816220","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:37136","multi.service_calls":0,"queuetimems":24918,"class":"HRegionServer","multi.mutations":1} 2022-03-07 17:28:38,158 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646673818157","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:37136","multi.service_calls":0,"queuetimems":26855,"class":"HRegionServer","multi.mutations":3} 2022-03-07 17:28:43,697 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646673823697","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:56184","multi.service_calls":0,"queuetimems":32394,"class":"HRegionServer","multi.mutations":2} 2022-03-07 17:28:51,391 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646673831391","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:56184","multi.service_calls":0,"queuetimems":40088,"class":"HRegionServer","multi.mutations":1} 2022-03-07 17:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=369504, hits=361394, hitRatio=97.81%, , cachingAccesses=361524, cachingHits=361085, cachingHitsRatio=99.88%, evictions=22246, evicted=0, evictedPerRun=0.0 2022-03-07 17:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-07 17:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 17:32:18,926 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646674038926","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:36934","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 17:32:19,701 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646674039700","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:37170","multi.service_calls":0,"queuetimems":774,"class":"HRegionServer","multi.mutations":1} 2022-03-07 17:33:24,107 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646673804105","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 4 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:56184","multi.service_calls":0,"queuetimems":20741,"class":"HRegionServer","multi.mutations":4} 2022-03-07 17:33:36,221 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646674116220","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:56206","multi.service_calls":0,"queuetimems":32855,"class":"HRegionServer","multi.mutations":1} 2022-03-07 17:33:41,795 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646673821794","responsesize":"16528","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 10 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":600001,"client":"172.31.0.219:56184","multi.service_calls":0,"queuetimems":30491,"class":"HRegionServer","multi.mutations":10} 2022-03-07 17:33:43,698 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646674123697","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:36946","multi.service_calls":0,"queuetimems":5539,"class":"HRegionServer","multi.mutations":1} 2022-03-07 17:33:51,392 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646674131391","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:56206","multi.service_calls":0,"queuetimems":48026,"class":"HRegionServer","multi.mutations":1} 2022-03-07 17:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=370242, hits=362132, hitRatio=97.81%, , cachingAccesses=362262, cachingHits=361823, cachingHitsRatio=99.88%, evictions=22276, evicted=0, evictedPerRun=0.0 2022-03-07 17:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 17:37:18,927 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646674338927","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:36990","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 17:37:19,702 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646674339701","responsesize":"8265","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 5 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:36990","multi.service_calls":0,"queuetimems":774,"class":"HRegionServer","multi.mutations":5} 2022-03-07 17:37:25,435 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646674045434","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 8 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:36934","multi.service_calls":0,"queuetimems":6508,"class":"HRegionServer","multi.mutations":8} 2022-03-07 17:37:39,714 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646673759712","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 5 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:56180","multi.service_calls":0,"queuetimems":20012,"class":"HRegionServer","multi.mutations":5} 2022-03-07 17:37:48,876 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646674068876","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:37170","multi.service_calls":0,"queuetimems":29950,"class":"HRegionServer","multi.mutations":2} 2022-03-07 17:38:03,366 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646674083365","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:37178","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 17:38:09,917 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646674089916","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:37178","multi.service_calls":0,"queuetimems":6551,"class":"HRegionServer","multi.mutations":5} 2022-03-07 17:38:24,108 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646674404107","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:37008","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 17:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=371039, hits=362929, hitRatio=97.81%, , cachingAccesses=363059, cachingHits=362620, cachingHitsRatio=99.88%, evictions=22306, evicted=0, evictedPerRun=0.0 2022-03-07 17:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 17:42:48,877 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646674668876","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:37066","multi.service_calls":0,"queuetimems":29173,"class":"HRegionServer","multi.mutations":1} 2022-03-07 17:43:03,367 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646674683367","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:37076","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 17:43:09,917 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646674689917","responsesize":"9917","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:37076","multi.service_calls":0,"queuetimems":6550,"class":"HRegionServer","multi.mutations":6} 2022-03-07 17:43:11,305 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646673791301","responsesize":"11578","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 7 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200004,"client":"172.30.41.155:37136","multi.service_calls":0,"queuetimems":7937,"class":"HRegionServer","multi.mutations":7} 2022-03-07 17:43:24,109 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646674704108","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:37234","multi.service_calls":0,"queuetimems":20741,"class":"HRegionServer","multi.mutations":1} 2022-03-07 17:43:36,191 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646673816190","responsesize":"6622","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200001,"client":"172.30.41.118:36896","multi.service_calls":0,"queuetimems":24889,"class":"HRegionServer","multi.mutations":4} 2022-03-07 17:43:36,222 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646674416221","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:37218","multi.service_calls":0,"queuetimems":12113,"class":"HRegionServer","multi.mutations":2} 2022-03-07 17:43:38,159 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646674118158","responsesize":"19835","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 12 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.31.0.219:56206","multi.service_calls":0,"queuetimems":1937,"class":"HRegionServer","multi.mutations":12} 2022-03-07 17:43:51,393 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646674431392","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:37008","multi.service_calls":0,"queuetimems":9597,"class":"HRegionServer","multi.mutations":3} 2022-03-07 17:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=371880, hits=363770, hitRatio=97.82%, , cachingAccesses=363900, cachingHits=363461, cachingHitsRatio=99.88%, evictions=22336, evicted=0, evictedPerRun=0.0 2022-03-07 17:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 17:47:18,930 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646674638929","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:37066","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 17:47:19,703 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646674639702","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:37230","multi.service_calls":0,"queuetimems":773,"class":"HRegionServer","multi.mutations":3} 2022-03-07 17:47:48,879 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646674968878","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:37144","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 17:48:03,368 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646674983368","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:56256","multi.service_calls":0,"queuetimems":14490,"class":"HRegionServer","multi.mutations":1} 2022-03-07 17:48:09,918 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646674989917","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:37144","multi.service_calls":0,"queuetimems":21039,"class":"HRegionServer","multi.mutations":1} 2022-03-07 17:48:11,306 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646674991305","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:37250","multi.service_calls":0,"queuetimems":22427,"class":"HRegionServer","multi.mutations":1} 2022-03-07 17:48:24,111 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646675004109","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.155:37250","multi.service_calls":0,"queuetimems":35231,"class":"HRegionServer","multi.mutations":1} 2022-03-07 17:48:36,192 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646675016191","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:37144","multi.service_calls":0,"queuetimems":47313,"class":"HRegionServer","multi.mutations":1} 2022-03-07 17:48:36,222 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646675016222","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:37144","multi.service_calls":0,"queuetimems":47344,"class":"HRegionServer","multi.mutations":1} 2022-03-07 17:48:38,160 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646675018159","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:37144","multi.service_calls":0,"queuetimems":49281,"class":"HRegionServer","multi.mutations":1} 2022-03-07 17:48:41,767 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646673821765","responsesize":"24797","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 15 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1500002,"client":"172.30.41.118:36896","multi.service_calls":0,"queuetimems":30463,"class":"HRegionServer","multi.mutations":15} 2022-03-07 17:48:41,797 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646674421795","responsesize":"21487","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 13 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.30.41.118:37008","multi.service_calls":0,"queuetimems":17688,"class":"HRegionServer","multi.mutations":13} 2022-03-07 17:48:43,699 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646674423698","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 5 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:37218","multi.service_calls":0,"queuetimems":19591,"class":"HRegionServer","multi.mutations":5} 2022-03-07 17:48:51,393 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646675031393","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:37144","multi.service_calls":0,"queuetimems":62515,"class":"HRegionServer","multi.mutations":1} 2022-03-07 17:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=372649, hits=364539, hitRatio=97.82%, , cachingAccesses=364669, cachingHits=364230, cachingHitsRatio=99.88%, evictions=22366, evicted=0, evictedPerRun=0.0 2022-03-07 17:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 17:52:25,437 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646674645435","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 7 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.31.0.219:56224","multi.service_calls":0,"queuetimems":6506,"class":"HRegionServer","multi.mutations":7} 2022-03-07 17:52:39,715 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646674659714","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 8 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:37230","multi.service_calls":0,"queuetimems":20785,"class":"HRegionServer","multi.mutations":8} 2022-03-07 17:52:48,879 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646675268879","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:37266","multi.service_calls":0,"queuetimems":29948,"class":"HRegionServer","multi.mutations":1} 2022-03-07 17:53:09,918 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646675289918","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:56300","multi.service_calls":0,"queuetimems":6548,"class":"HRegionServer","multi.mutations":1} 2022-03-07 17:53:43,732 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646675323731","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.31.0.219:56304","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 17:53:51,394 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646675331393","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:37270","multi.service_calls":0,"queuetimems":40086,"class":"HRegionServer","multi.mutations":1} 2022-03-07 17:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=373477, hits=365367, hitRatio=97.83%, , cachingAccesses=365497, cachingHits=365058, cachingHitsRatio=99.88%, evictions=22396, evicted=0, evictedPerRun=0.0 2022-03-07 17:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 17:57:18,932 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646675238931","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:37188","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 17:57:19,705 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646675239704","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:37188","multi.service_calls":0,"queuetimems":773,"class":"HRegionServer","multi.mutations":2} 2022-03-07 17:57:25,440 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646675545438","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300002,"client":"172.30.41.118:37240","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 17:57:48,881 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646675568879","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.155:37296","multi.service_calls":0,"queuetimems":23440,"class":"HRegionServer","multi.mutations":1} 2022-03-07 17:58:03,370 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646675283369","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":600001,"client":"172.30.41.118:37198","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 17:58:09,919 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646675589919","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:37296","multi.service_calls":0,"queuetimems":44481,"class":"HRegionServer","multi.mutations":1} 2022-03-07 17:58:11,308 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646675291306","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 4 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.118:37198","multi.service_calls":0,"queuetimems":7936,"class":"HRegionServer","multi.mutations":4} 2022-03-07 17:58:36,193 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646675316192","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:37198","multi.service_calls":0,"queuetimems":24885,"class":"HRegionServer","multi.mutations":3} 2022-03-07 17:58:38,161 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646675318160","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 9 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:37198","multi.service_calls":0,"queuetimems":26853,"class":"HRegionServer","multi.mutations":9} 2022-03-07 17:58:41,768 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646675321767","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 9 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:56300","multi.service_calls":0,"queuetimems":38397,"class":"HRegionServer","multi.mutations":9} 2022-03-07 17:58:51,395 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646675631394","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:37240","multi.service_calls":0,"queuetimems":71678,"class":"HRegionServer","multi.mutations":1} 2022-03-07 18:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=374317, hits=366207, hitRatio=97.83%, , cachingAccesses=366337, cachingHits=365898, cachingHitsRatio=99.88%, evictions=22426, evicted=0, evictedPerRun=0.0 2022-03-07 18:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 18:02:18,933 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646675838933","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:37290","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 18:02:19,706 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646675839705","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.30.41.155:37332","multi.service_calls":0,"queuetimems":742,"class":"HRegionServer","multi.mutations":2} 2022-03-07 18:02:39,717 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646675559716","responsesize":"16528","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 10 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":600001,"client":"172.30.41.118:37240","multi.service_calls":0,"queuetimems":14278,"class":"HRegionServer","multi.mutations":10} 2022-03-07 18:02:48,882 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646675868881","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:37334","multi.service_calls":0,"queuetimems":29948,"class":"HRegionServer","multi.mutations":2} 2022-03-07 18:03:09,920 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646675889919","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.30.41.155:37338","multi.service_calls":0,"queuetimems":6517,"class":"HRegionServer","multi.mutations":2} 2022-03-07 18:03:24,113 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646675304111","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.155:37270","multi.service_calls":0,"queuetimems":20741,"class":"HRegionServer","multi.mutations":3} 2022-03-07 18:03:36,194 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646675916193","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:56362","multi.service_calls":0,"queuetimems":24880,"class":"HRegionServer","multi.mutations":1} 2022-03-07 18:03:36,224 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646675316222","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 7 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.155:37270","multi.service_calls":0,"queuetimems":32852,"class":"HRegionServer","multi.mutations":7} 2022-03-07 18:03:38,162 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646675918161","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:37298","multi.service_calls":0,"queuetimems":26848,"class":"HRegionServer","multi.mutations":1} 2022-03-07 18:03:41,798 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646675321797","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 6 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:56300","multi.service_calls":0,"queuetimems":30490,"class":"HRegionServer","multi.mutations":6} 2022-03-07 18:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=375105, hits=366995, hitRatio=97.84%, , cachingAccesses=367125, cachingHits=366686, cachingHitsRatio=99.88%, evictions=22456, evicted=0, evictedPerRun=0.0 2022-03-07 18:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 18:07:18,935 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646676138934","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:37370","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 18:07:19,707 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646676139706","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:37370","multi.service_calls":0,"queuetimems":772,"class":"HRegionServer","multi.mutations":1} 2022-03-07 18:07:39,717 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646676159717","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:37370","multi.service_calls":0,"queuetimems":20783,"class":"HRegionServer","multi.mutations":1} 2022-03-07 18:07:48,882 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646676168882","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:37342","multi.service_calls":0,"queuetimems":29948,"class":"HRegionServer","multi.mutations":3} 2022-03-07 18:08:03,372 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646675883371","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 6 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:37298","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":6} 2022-03-07 18:08:09,921 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646676189921","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:37378","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 18:08:24,114 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646676204114","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:37352","multi.service_calls":0,"queuetimems":14193,"class":"HRegionServer","multi.mutations":3} 2022-03-07 18:08:36,194 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646676216194","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:37378","multi.service_calls":0,"queuetimems":12080,"class":"HRegionServer","multi.mutations":1} 2022-03-07 18:08:43,733 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646675623732","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 7 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:37240","multi.service_calls":0,"queuetimems":64016,"class":"HRegionServer","multi.mutations":7} 2022-03-07 18:08:51,396 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646675931395","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 5 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:37298","multi.service_calls":0,"queuetimems":40082,"class":"HRegionServer","multi.mutations":5} 2022-03-07 18:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=375884, hits=367774, hitRatio=97.84%, , cachingAccesses=367904, cachingHits=367465, cachingHitsRatio=99.88%, evictions=22486, evicted=0, evictedPerRun=0.0 2022-03-07 18:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 18:12:25,443 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646675845440","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 8 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900003,"client":"172.30.41.118:37290","multi.service_calls":0,"queuetimems":6507,"class":"HRegionServer","multi.mutations":8} 2022-03-07 18:12:48,883 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646676468882","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:37392","multi.service_calls":0,"queuetimems":29946,"class":"HRegionServer","multi.mutations":1} 2022-03-07 18:13:11,309 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646675891308","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 7 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:37298","multi.service_calls":0,"queuetimems":7937,"class":"HRegionServer","multi.mutations":7} 2022-03-07 18:13:36,225 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646676216224","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:56378","multi.service_calls":0,"queuetimems":26303,"class":"HRegionServer","multi.mutations":5} 2022-03-07 18:13:51,396 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646676531396","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:37394","multi.service_calls":0,"queuetimems":48023,"class":"HRegionServer","multi.mutations":2} 2022-03-07 18:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=376715, hits=368605, hitRatio=97.85%, , cachingAccesses=368735, cachingHits=368296, cachingHitsRatio=99.88%, evictions=22516, evicted=0, evictedPerRun=0.0 2022-03-07 18:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 18:17:19,708 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646676439707","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:37392","multi.service_calls":0,"queuetimems":771,"class":"HRegionServer","multi.mutations":2} 2022-03-07 18:17:25,444 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646676745444","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:37488","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 18:18:03,374 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646676483373","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:37418","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 18:18:11,311 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646676791310","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:37498","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 18:18:24,116 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646676504115","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:37418","multi.service_calls":0,"queuetimems":20742,"class":"HRegionServer","multi.mutations":4} 2022-03-07 18:18:36,196 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646676516195","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 4 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.31.0.219:56398","multi.service_calls":0,"queuetimems":32821,"class":"HRegionServer","multi.mutations":4} 2022-03-07 18:18:36,225 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646676816225","responsesize":"11569","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 7 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:37498","multi.service_calls":0,"queuetimems":24915,"class":"HRegionServer","multi.mutations":7} 2022-03-07 18:18:43,734 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646676523733","responsesize":"11572","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:56398","multi.service_calls":0,"queuetimems":40359,"class":"HRegionServer","multi.mutations":7} 2022-03-07 18:18:51,397 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646676831396","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:56420","multi.service_calls":0,"queuetimems":40085,"class":"HRegionServer","multi.mutations":1} 2022-03-07 18:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=377482, hits=369372, hitRatio=97.85%, , cachingAccesses=369502, cachingHits=369063, cachingHitsRatio=99.88%, evictions=22546, evicted=0, evictedPerRun=0.0 2022-03-07 18:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 18:22:18,937 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646676438936","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:37412","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 18:22:25,445 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646677045445","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:37540","multi.service_calls":0,"queuetimems":5736,"class":"HRegionServer","multi.mutations":1} 2022-03-07 18:22:39,719 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646676459718","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 4 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:56396","multi.service_calls":0,"queuetimems":20782,"class":"HRegionServer","multi.mutations":4} 2022-03-07 18:23:03,375 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646677083375","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:37562","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 18:23:09,923 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646676489921","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 7 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:37394","multi.service_calls":0,"queuetimems":6548,"class":"HRegionServer","multi.mutations":7} 2022-03-07 18:23:11,312 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646677091311","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:37460","multi.service_calls":0,"queuetimems":7936,"class":"HRegionServer","multi.mutations":1} 2022-03-07 18:23:36,226 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646677116225","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.30.41.155:37462","multi.service_calls":0,"queuetimems":32820,"class":"HRegionServer","multi.mutations":2} 2022-03-07 18:23:38,163 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646676218162","responsesize":"14882","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 9 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200001,"client":"172.30.41.155:37378","multi.service_calls":0,"queuetimems":14048,"class":"HRegionServer","multi.mutations":9} 2022-03-07 18:23:41,770 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646675921768","responsesize":"26449","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 16 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1500002,"client":"172.31.0.219:56362","multi.service_calls":0,"queuetimems":30455,"class":"HRegionServer","multi.mutations":16} 2022-03-07 18:23:41,800 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646676221798","responsesize":"23142","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 14 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.118:37352","multi.service_calls":0,"queuetimems":17684,"class":"HRegionServer","multi.mutations":14} 2022-03-07 18:23:51,398 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646677131397","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:56432","multi.service_calls":0,"queuetimems":40085,"class":"HRegionServer","multi.mutations":1} 2022-03-07 18:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=378310, hits=370200, hitRatio=97.86%, , cachingAccesses=370330, cachingHits=369891, cachingHitsRatio=99.88%, evictions=22576, evicted=0, evictedPerRun=0.0 2022-03-07 18:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 18:27:19,710 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646677039709","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:37540","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":5} 2022-03-07 18:27:25,447 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646677345445","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:37608","multi.service_calls":0,"queuetimems":6507,"class":"HRegionServer","multi.mutations":2} 2022-03-07 18:27:48,884 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646676768883","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:37412","multi.service_calls":0,"queuetimems":23439,"class":"HRegionServer","multi.mutations":3} 2022-03-07 18:28:03,376 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646677383375","responsesize":"6613","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:56448","multi.service_calls":0,"queuetimems":44437,"class":"HRegionServer","multi.mutations":4} 2022-03-07 18:28:11,312 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646677391312","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:37486","multi.service_calls":0,"queuetimems":52374,"class":"HRegionServer","multi.mutations":2} 2022-03-07 18:28:24,117 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646677104117","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:37562","multi.service_calls":0,"queuetimems":20742,"class":"HRegionServer","multi.mutations":4} 2022-03-07 18:28:36,197 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646677116196","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 6 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:37460","multi.service_calls":0,"queuetimems":32821,"class":"HRegionServer","multi.mutations":6} 2022-03-07 18:28:38,164 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646677418164","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:37494","multi.service_calls":0,"queuetimems":1937,"class":"HRegionServer","multi.mutations":1} 2022-03-07 18:28:43,735 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646677123734","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:56432","multi.service_calls":0,"queuetimems":40359,"class":"HRegionServer","multi.mutations":3} 2022-03-07 18:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=379090, hits=370980, hitRatio=97.86%, , cachingAccesses=371110, cachingHits=370671, cachingHitsRatio=99.88%, evictions=22606, evicted=0, evictedPerRun=0.0 2022-03-07 18:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-07 18:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 18:32:18,939 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646677338938","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:37608","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-07 18:33:03,377 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646677683376","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:37686","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 18:33:09,923 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646677389923","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:37608","multi.service_calls":0,"queuetimems":30203,"class":"HRegionServer","multi.mutations":2} 2022-03-07 18:33:11,313 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646677691313","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:37522","multi.service_calls":0,"queuetimems":7936,"class":"HRegionServer","multi.mutations":1} 2022-03-07 18:33:24,119 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646677704117","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.155:37522","multi.service_calls":0,"queuetimems":20740,"class":"HRegionServer","multi.mutations":1} 2022-03-07 18:33:36,228 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646677416227","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:37624","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-07 18:33:38,165 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646677718164","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:37522","multi.service_calls":0,"queuetimems":34787,"class":"HRegionServer","multi.mutations":1} 2022-03-07 18:33:41,801 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646677421800","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 9 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:56454","multi.service_calls":0,"queuetimems":5573,"class":"HRegionServer","multi.mutations":9} 2022-03-07 18:33:43,736 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646677723735","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:37522","multi.service_calls":0,"queuetimems":40358,"class":"HRegionServer","multi.mutations":1} 2022-03-07 18:33:51,400 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646677431398","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.155:37494","multi.service_calls":0,"queuetimems":15171,"class":"HRegionServer","multi.mutations":2} 2022-03-07 18:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=379802, hits=371692, hitRatio=97.86%, , cachingAccesses=371822, cachingHits=371383, cachingHitsRatio=99.88%, evictions=22636, evicted=0, evictedPerRun=0.0 2022-03-07 18:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 18:37:19,712 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646677639711","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:37678","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-07 18:37:25,448 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646677645447","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 9 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:37678","multi.service_calls":0,"queuetimems":5736,"class":"HRegionServer","multi.mutations":9} 2022-03-07 18:37:39,721 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646677359720","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:37486","multi.service_calls":0,"queuetimems":20782,"class":"HRegionServer","multi.mutations":4} 2022-03-07 18:38:03,378 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646677983377","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:56486","multi.service_calls":0,"queuetimems":38239,"class":"HRegionServer","multi.mutations":1} 2022-03-07 18:38:09,924 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646677989923","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:56486","multi.service_calls":0,"queuetimems":44419,"class":"HRegionServer","multi.mutations":1} 2022-03-07 18:38:11,314 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646677991313","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:37748","multi.service_calls":0,"queuetimems":49969,"class":"HRegionServer","multi.mutations":1} 2022-03-07 18:38:24,120 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646678004119","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:56486","multi.service_calls":0,"queuetimems":62357,"class":"HRegionServer","multi.mutations":1} 2022-03-07 18:38:36,198 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646677716197","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 6 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:37686","multi.service_calls":0,"queuetimems":32820,"class":"HRegionServer","multi.mutations":6} 2022-03-07 18:38:36,228 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646678016228","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:37748","multi.service_calls":0,"queuetimems":68662,"class":"HRegionServer","multi.mutations":1} 2022-03-07 18:38:38,165 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646678018165","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:37748","multi.service_calls":0,"queuetimems":70625,"class":"HRegionServer","multi.mutations":1} 2022-03-07 18:38:41,772 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646677421770","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 8 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:37624","multi.service_calls":0,"queuetimems":5543,"class":"HRegionServer","multi.mutations":8} 2022-03-07 18:38:41,801 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646678021801","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:37538","multi.service_calls":0,"queuetimems":73554,"class":"HRegionServer","multi.mutations":1} 2022-03-07 18:38:43,736 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646678023736","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:37538","multi.service_calls":0,"queuetimems":74803,"class":"HRegionServer","multi.mutations":1} 2022-03-07 18:38:51,400 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646678031400","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:37538","multi.service_calls":0,"queuetimems":76940,"class":"HRegionServer","multi.mutations":1} 2022-03-07 18:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=380562, hits=372452, hitRatio=97.87%, , cachingAccesses=372582, cachingHits=372143, cachingHitsRatio=99.88%, evictions=22666, evicted=0, evictedPerRun=0.0 2022-03-07 18:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 18:42:18,941 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646677938940","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:37538","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 18:42:48,885 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646677668884","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:56468","multi.service_calls":0,"queuetimems":29173,"class":"HRegionServer","multi.mutations":5} 2022-03-07 18:43:11,314 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646678291314","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:37816","multi.service_calls":0,"queuetimems":45865,"class":"HRegionServer","multi.mutations":1} 2022-03-07 18:43:24,121 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646678304121","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:37830","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 18:43:41,772 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646678321772","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:37830","multi.service_calls":0,"queuetimems":5574,"class":"HRegionServer","multi.mutations":1} 2022-03-07 18:43:51,401 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646678331401","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:37830","multi.service_calls":0,"queuetimems":15203,"class":"HRegionServer","multi.mutations":1} 2022-03-07 18:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=381234, hits=373124, hitRatio=97.87%, , cachingAccesses=373254, cachingHits=372815, cachingHitsRatio=99.88%, evictions=22696, evicted=0, evictedPerRun=0.0 2022-03-07 18:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 18:47:19,714 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646678239713","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 4 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:37816","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-07 18:47:39,722 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646678259721","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:56506","multi.service_calls":0,"queuetimems":20008,"class":"HRegionServer","multi.mutations":2} 2022-03-07 18:47:48,886 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646678568885","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:37576","multi.service_calls":0,"queuetimems":29943,"class":"HRegionServer","multi.mutations":1} 2022-03-07 18:48:09,925 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646678289924","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:37554","multi.service_calls":0,"queuetimems":44475,"class":"HRegionServer","multi.mutations":3} 2022-03-07 18:48:11,315 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646678591315","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:37892","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 18:48:24,123 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646678604121","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.155:37578","multi.service_calls":0,"queuetimems":12806,"class":"HRegionServer","multi.mutations":1} 2022-03-07 18:48:36,199 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646678316198","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:56510","multi.service_calls":0,"queuetimems":12077,"class":"HRegionServer","multi.mutations":3} 2022-03-07 18:48:36,229 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646678316228","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 5 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:37562","multi.service_calls":0,"queuetimems":12107,"class":"HRegionServer","multi.mutations":5} 2022-03-07 18:48:38,166 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646678318165","responsesize":"18180","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 11 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":600001,"client":"172.31.0.219:56510","multi.service_calls":0,"queuetimems":14044,"class":"HRegionServer","multi.mutations":11} 2022-03-07 18:48:41,773 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646678621773","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:37902","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 18:48:43,737 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646678323736","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:37562","multi.service_calls":0,"queuetimems":7538,"class":"HRegionServer","multi.mutations":6} 2022-03-07 18:49:46,100 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3605ms No GCs detected 2022-03-07 18:49:58,384 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 6229ms No GCs detected 2022-03-07 18:50:00,737 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1853ms No GCs detected 2022-03-07 18:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=381997, hits=373887, hitRatio=97.88%, , cachingAccesses=374017, cachingHits=373578, cachingHitsRatio=99.88%, evictions=22726, evicted=0, evictedPerRun=0.0 2022-03-07 18:51:48,028 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 18:52:19,716 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646678839715","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:37950","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 18:52:25,451 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646678245449","responsesize":"19835","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 12 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Dev ","processingtimems":900002,"client":"172.30.41.118:37816","multi.service_calls":0,"queuetimems":5736,"class":"HRegionServer","multi.mutations":12} 2022-03-07 18:53:11,316 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646678891316","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:37590","multi.service_calls":0,"queuetimems":22429,"class":"HRegionServer","multi.mutations":1} 2022-03-07 18:53:24,124 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646678904123","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:37590","multi.service_calls":0,"queuetimems":35236,"class":"HRegionServer","multi.mutations":1} 2022-03-07 18:53:43,738 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646678923737","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:37594","multi.service_calls":0,"queuetimems":7537,"class":"HRegionServer","multi.mutations":3} 2022-03-07 18:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=382651, hits=374541, hitRatio=97.88%, , cachingAccesses=374671, cachingHits=374232, cachingHitsRatio=99.88%, evictions=22756, evicted=0, evictedPerRun=0.0 2022-03-07 18:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 18:57:39,723 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646678859722","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 4 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:37950","multi.service_calls":0,"queuetimems":20007,"class":"HRegionServer","multi.mutations":4} 2022-03-07 18:57:48,887 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646678868887","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.31.0.219:56538","multi.service_calls":0,"queuetimems":29171,"class":"HRegionServer","multi.mutations":3} 2022-03-07 18:58:03,380 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646678283378","responsesize":"23142","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 14 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.31.0.219:56506","multi.service_calls":0,"queuetimems":43665,"class":"HRegionServer","multi.mutations":14} 2022-03-07 18:58:11,317 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646679191316","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:37608","multi.service_calls":0,"queuetimems":45863,"class":"HRegionServer","multi.mutations":1} 2022-03-07 18:58:24,125 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646679204124","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:37608","multi.service_calls":0,"queuetimems":6989,"class":"HRegionServer","multi.mutations":1} 2022-03-07 18:58:36,201 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646678916200","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:37968","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-07 18:58:36,231 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646678916230","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:37594","multi.service_calls":0,"queuetimems":30,"class":"HRegionServer","multi.mutations":3} 2022-03-07 18:58:41,803 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646678321801","responsesize":"24794","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 15 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.118:37830","multi.service_calls":0,"queuetimems":5603,"class":"HRegionServer","multi.mutations":15} 2022-03-07 19:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=383316, hits=375206, hitRatio=97.88%, , cachingAccesses=375336, cachingHits=374897, cachingHitsRatio=99.88%, evictions=22786, evicted=0, evictedPerRun=0.0 2022-03-07 19:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 19:02:48,889 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646679468888","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:56584","multi.service_calls":0,"queuetimems":9164,"class":"HRegionServer","multi.mutations":1} 2022-03-07 19:03:03,381 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646679483380","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:37624","multi.service_calls":0,"queuetimems":23656,"class":"HRegionServer","multi.mutations":1} 2022-03-07 19:03:09,926 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646678889925","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:37950","multi.service_calls":0,"queuetimems":21038,"class":"HRegionServer","multi.mutations":3} 2022-03-07 19:03:11,318 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646679491317","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:38064","multi.service_calls":0,"queuetimems":22429,"class":"HRegionServer","multi.mutations":1} 2022-03-07 19:03:43,739 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646679223738","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 5 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:37608","multi.service_calls":0,"queuetimems":84021,"class":"HRegionServer","multi.mutations":5} 2022-03-07 19:03:51,404 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646678631401","responsesize":"23142","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 14 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.118:37902","multi.service_calls":0,"queuetimems":9628,"class":"HRegionServer","multi.mutations":14} 2022-03-07 19:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=383893, hits=375783, hitRatio=97.89%, , cachingAccesses=375913, cachingHits=375474, cachingHitsRatio=99.88%, evictions=22816, evicted=0, evictedPerRun=0.0 2022-03-07 19:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 19:07:18,945 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646678538942","responsesize":"31405","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 19 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Dev ","processingtimems":1500003,"client":"172.30.41.118:37874","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":19} 2022-03-07 19:07:25,455 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646679145452","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 8 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900003,"client":"172.30.41.118:38012","multi.service_calls":0,"queuetimems":5735,"class":"HRegionServer","multi.mutations":8} 2022-03-07 19:07:39,725 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646679459724","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 5 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:38064","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":5} 2022-03-07 19:08:11,318 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646679791318","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:56614","multi.service_calls":0,"queuetimems":22428,"class":"HRegionServer","multi.mutations":1} 2022-03-07 19:08:24,128 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646679504126","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.155:37628","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-07 19:08:36,203 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646679516201","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:56592","multi.service_calls":0,"queuetimems":12075,"class":"HRegionServer","multi.mutations":3} 2022-03-07 19:08:38,168 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646678918166","responsesize":"19838","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 12 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.118:37968","multi.service_calls":0,"queuetimems":1966,"class":"HRegionServer","multi.mutations":12} 2022-03-07 19:08:41,776 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646678921773","responsesize":"13230","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 8 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200002,"client":"172.31.0.219:56542","multi.service_calls":0,"queuetimems":5572,"class":"HRegionServer","multi.mutations":8} 2022-03-07 19:08:41,804 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646679521803","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:38078","multi.service_calls":0,"queuetimems":5601,"class":"HRegionServer","multi.mutations":2} 2022-03-07 19:08:43,740 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646679823739","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:56614","multi.service_calls":0,"queuetimems":54849,"class":"HRegionServer","multi.mutations":1} 2022-03-07 19:08:51,404 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646679831404","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:56614","multi.service_calls":0,"queuetimems":62514,"class":"HRegionServer","multi.mutations":1} 2022-03-07 19:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=384640, hits=376530, hitRatio=97.89%, , cachingAccesses=376660, cachingHits=376221, cachingHitsRatio=99.88%, evictions=22846, evicted=0, evictedPerRun=0.0 2022-03-07 19:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 19:12:18,947 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646680038946","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:37648","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 19:12:19,719 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646679139717","responsesize":"11578","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 7 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200002,"client":"172.30.41.118:38012","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":7} 2022-03-07 19:12:25,457 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646680045455","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.155:37648","multi.service_calls":0,"queuetimems":6509,"class":"HRegionServer","multi.mutations":1} 2022-03-07 19:12:39,726 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646680059726","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:38170","multi.service_calls":0,"queuetimems":20780,"class":"HRegionServer","multi.mutations":1} 2022-03-07 19:12:48,891 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646679768890","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:37636","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 19:13:03,383 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646679783382","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 6 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:38116","multi.service_calls":0,"queuetimems":14492,"class":"HRegionServer","multi.mutations":6} 2022-03-07 19:13:09,928 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646679789926","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:37636","multi.service_calls":0,"queuetimems":21036,"class":"HRegionServer","multi.mutations":3} 2022-03-07 19:13:11,319 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646680091319","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:37648","multi.service_calls":0,"queuetimems":31593,"class":"HRegionServer","multi.mutations":1} 2022-03-07 19:13:36,232 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646679516231","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:38078","multi.service_calls":0,"queuetimems":12105,"class":"HRegionServer","multi.mutations":6} 2022-03-07 19:13:41,806 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646680121805","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300001,"client":"172.30.41.118:38194","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 19:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=385340, hits=377230, hitRatio=97.90%, , cachingAccesses=377360, cachingHits=376921, cachingHitsRatio=99.88%, evictions=22876, evicted=0, evictedPerRun=0.0 2022-03-07 19:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 19:17:25,458 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646680345457","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:37664","multi.service_calls":0,"queuetimems":6509,"class":"HRegionServer","multi.mutations":1} 2022-03-07 19:18:03,383 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646680383383","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:56644","multi.service_calls":0,"queuetimems":44435,"class":"HRegionServer","multi.mutations":1} 2022-03-07 19:18:24,131 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646680104128","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 9 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.118:38170","multi.service_calls":0,"queuetimems":44402,"class":"HRegionServer","multi.mutations":9} 2022-03-07 19:18:36,204 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646680116203","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 4 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:56628","multi.service_calls":0,"queuetimems":77256,"class":"HRegionServer","multi.mutations":4} 2022-03-07 19:18:36,234 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646680416233","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:38262","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 19:18:41,777 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646680121776","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:38194","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 19:18:41,807 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646680421806","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:38262","multi.service_calls":0,"queuetimems":5573,"class":"HRegionServer","multi.mutations":1} 2022-03-07 19:18:51,405 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646680131404","responsesize":"11572","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:38194","multi.service_calls":0,"queuetimems":9628,"class":"HRegionServer","multi.mutations":7} 2022-03-07 19:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=386131, hits=378021, hitRatio=97.90%, , cachingAccesses=378151, cachingHits=377712, cachingHitsRatio=99.88%, evictions=22906, evicted=0, evictedPerRun=0.0 2022-03-07 19:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 19:22:19,720 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646680339719","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:56644","multi.service_calls":0,"queuetimems":771,"class":"HRegionServer","multi.mutations":3} 2022-03-07 19:22:25,459 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646680645459","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:37682","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 19:23:11,320 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646680391319","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:56644","multi.service_calls":0,"queuetimems":51599,"class":"HRegionServer","multi.mutations":2} 2022-03-07 19:23:24,132 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646680704131","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:38318","multi.service_calls":0,"queuetimems":20717,"class":"HRegionServer","multi.mutations":2} 2022-03-07 19:23:36,204 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646680716204","responsesize":"8265","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 5 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:38320","multi.service_calls":0,"queuetimems":32820,"class":"HRegionServer","multi.mutations":5} 2022-03-07 19:23:41,807 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646680721807","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:37684","multi.service_calls":0,"queuetimems":38422,"class":"HRegionServer","multi.mutations":1} 2022-03-07 19:23:41,809 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646680721808","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.30.41.118:38334","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 19:23:43,741 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646680123740","responsesize":"14879","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 9 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:38194","multi.service_calls":0,"queuetimems":1935,"class":"HRegionServer","multi.mutations":9} 2022-03-07 19:23:51,406 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646680731406","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300000,"client":"172.30.41.118:38332","multi.service_calls":0,"queuetimems":9591,"class":"HRegionServer","multi.mutations":1} 2022-03-07 19:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=386960, hits=378850, hitRatio=97.90%, , cachingAccesses=378980, cachingHits=378541, cachingHitsRatio=99.88%, evictions=22936, evicted=0, evictedPerRun=0.0 2022-03-07 19:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 19:27:18,949 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646680338948","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 5 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:38242","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":5} 2022-03-07 19:27:19,722 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646680939721","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:37700","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 19:27:25,462 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646680945461","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:38386","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-07 19:27:39,728 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646680359726","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 8 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.155:37664","multi.service_calls":0,"queuetimems":20778,"class":"HRegionServer","multi.mutations":8} 2022-03-07 19:28:03,385 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646680683384","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 5 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:38320","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":5} 2022-03-07 19:28:09,929 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646680389928","responsesize":"18183","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 11 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.31.0.219:56644","multi.service_calls":0,"queuetimems":50208,"class":"HRegionServer","multi.mutations":11} 2022-03-07 19:28:38,170 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646680118169","responsesize":"14882","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 9 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200001,"client":"172.30.41.155:37648","multi.service_calls":0,"queuetimems":58443,"class":"HRegionServer","multi.mutations":9} 2022-03-07 19:28:41,810 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646681021809","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:37706","multi.service_calls":0,"queuetimems":5604,"class":"HRegionServer","multi.mutations":1} 2022-03-07 19:28:51,407 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646681031406","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:37706","multi.service_calls":0,"queuetimems":15201,"class":"HRegionServer","multi.mutations":1} 2022-03-07 19:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=387709, hits=379599, hitRatio=97.91%, , cachingAccesses=379729, cachingHits=379290, cachingHitsRatio=99.88%, evictions=22966, evicted=0, evictedPerRun=0.0 2022-03-07 19:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-07 19:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 19:32:18,951 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646681238950","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:38456","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 19:32:19,723 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646681239722","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:37718","multi.service_calls":0,"queuetimems":772,"class":"HRegionServer","multi.mutations":1} 2022-03-07 19:32:48,893 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646680368891","responsesize":"18186","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 11 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.118:38242","multi.service_calls":0,"queuetimems":29943,"class":"HRegionServer","multi.mutations":11} 2022-03-07 19:33:09,930 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646681289929","responsesize":"6613","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:37720","multi.service_calls":0,"queuetimems":6542,"class":"HRegionServer","multi.mutations":4} 2022-03-07 19:33:11,321 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646680991320","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:37700","multi.service_calls":0,"queuetimems":45859,"class":"HRegionServer","multi.mutations":3} 2022-03-07 19:33:36,206 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646681016205","responsesize":"11572","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 7 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:38404","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":7} 2022-03-07 19:33:36,235 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646680716234","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 8 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:38320","multi.service_calls":0,"queuetimems":12103,"class":"HRegionServer","multi.mutations":8} 2022-03-07 19:33:38,171 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646681318171","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:38478","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 19:33:41,811 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646681321810","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:56716","multi.service_calls":0,"queuetimems":38423,"class":"HRegionServer","multi.mutations":1} 2022-03-07 19:33:43,742 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646681023741","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 9 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:56684","multi.service_calls":0,"queuetimems":7535,"class":"HRegionServer","multi.mutations":9} 2022-03-07 19:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=388515, hits=380405, hitRatio=97.91%, , cachingAccesses=380535, cachingHits=380096, cachingHitsRatio=99.88%, evictions=22996, evicted=0, evictedPerRun=0.0 2022-03-07 19:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 19:37:18,952 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646681538952","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:37736","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 19:37:19,724 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646681539723","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:37736","multi.service_calls":0,"queuetimems":771,"class":"HRegionServer","multi.mutations":2} 2022-03-07 19:37:25,463 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646681245462","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:56708","multi.service_calls":0,"queuetimems":6511,"class":"HRegionServer","multi.mutations":3} 2022-03-07 19:37:39,731 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646681259728","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600003,"client":"172.30.41.155:37718","multi.service_calls":0,"queuetimems":20778,"class":"HRegionServer","multi.mutations":3} 2022-03-07 19:38:11,322 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646681591321","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:56742","multi.service_calls":0,"queuetimems":52369,"class":"HRegionServer","multi.mutations":1} 2022-03-07 19:38:36,208 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646681616207","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:38530","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 19:38:38,172 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646681618171","responsesize":"9917","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:38530","multi.service_calls":0,"queuetimems":1964,"class":"HRegionServer","multi.mutations":6} 2022-03-07 19:38:41,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646681021807","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:56684","multi.service_calls":0,"queuetimems":5601,"class":"HRegionServer","multi.mutations":3} 2022-03-07 19:38:44,132 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1873ms No GCs detected 2022-03-07 19:38:44,132 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646681623742","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300390,"client":"172.31.0.219:56748","multi.service_calls":0,"queuetimems":7534,"class":"HRegionServer","multi.mutations":1} 2022-03-07 19:38:45,869 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1236ms No GCs detected 2022-03-07 19:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=389272, hits=381162, hitRatio=97.92%, , cachingAccesses=381292, cachingHits=380853, cachingHitsRatio=99.88%, evictions=23026, evicted=0, evictedPerRun=0.0 2022-03-07 19:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 19:42:18,954 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646681838953","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:38568","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 19:42:25,465 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646681845463","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.118:38568","multi.service_calls":0,"queuetimems":6510,"class":"HRegionServer","multi.mutations":1} 2022-03-07 19:43:03,388 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646681283386","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 8 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.118:38464","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":8} 2022-03-07 19:43:24,134 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646681004132","responsesize":"23142","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 14 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.118:38386","multi.service_calls":0,"queuetimems":58671,"class":"HRegionServer","multi.mutations":14} 2022-03-07 19:43:38,173 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646681918172","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:56786","multi.service_calls":0,"queuetimems":26849,"class":"HRegionServer","multi.mutations":1} 2022-03-07 19:43:41,809 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646681921808","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:56786","multi.service_calls":0,"queuetimems":30485,"class":"HRegionServer","multi.mutations":1} 2022-03-07 19:43:41,811 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646681621811","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:37742","multi.service_calls":0,"queuetimems":5604,"class":"HRegionServer","multi.mutations":2} 2022-03-07 19:43:44,133 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646681924132","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:37758","multi.service_calls":0,"queuetimems":7923,"class":"HRegionServer","multi.mutations":1} 2022-03-07 19:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=390045, hits=381935, hitRatio=97.92%, , cachingAccesses=382065, cachingHits=381626, cachingHitsRatio=99.89%, evictions=23056, evicted=0, evictedPerRun=0.0 2022-03-07 19:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 19:47:18,955 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646682138955","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:37772","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 19:47:25,465 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646682145465","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:38620","multi.service_calls":0,"queuetimems":6510,"class":"HRegionServer","multi.mutations":1} 2022-03-07 19:47:39,732 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646681859731","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:37756","multi.service_calls":0,"queuetimems":20778,"class":"HRegionServer","multi.mutations":5} 2022-03-07 19:47:48,894 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646681568893","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 8 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:38516","multi.service_calls":0,"queuetimems":29941,"class":"HRegionServer","multi.mutations":8} 2022-03-07 19:48:03,389 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646682183389","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:38624","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":2} 2022-03-07 19:48:36,209 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646681916208","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 5 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:37758","multi.service_calls":0,"queuetimems":24885,"class":"HRegionServer","multi.mutations":5} 2022-03-07 19:48:36,237 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646681616236","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 7 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:37742","multi.service_calls":0,"queuetimems":29,"class":"HRegionServer","multi.mutations":7} 2022-03-07 19:48:41,810 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646682221809","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:38624","multi.service_calls":0,"queuetimems":17674,"class":"HRegionServer","multi.mutations":1} 2022-03-07 19:48:44,133 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646682224133","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:38624","multi.service_calls":0,"queuetimems":19998,"class":"HRegionServer","multi.mutations":1} 2022-03-07 19:48:51,408 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646681331407","responsesize":"21490","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 13 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200001,"client":"172.31.0.219:56716","multi.service_calls":0,"queuetimems":48020,"class":"HRegionServer","multi.mutations":13} 2022-03-07 19:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=390781, hits=382671, hitRatio=97.92%, , cachingAccesses=382801, cachingHits=382362, cachingHitsRatio=99.89%, evictions=23086, evicted=0, evictedPerRun=0.0 2022-03-07 19:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 19:52:18,956 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646682438956","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:38676","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 19:52:19,725 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646681839724","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:37756","multi.service_calls":0,"queuetimems":771,"class":"HRegionServer","multi.mutations":3} 2022-03-07 19:52:39,733 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646682459733","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:37790","multi.service_calls":0,"queuetimems":20777,"class":"HRegionServer","multi.mutations":2} 2022-03-07 19:52:48,895 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646682468894","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:37790","multi.service_calls":0,"queuetimems":29938,"class":"HRegionServer","multi.mutations":2} 2022-03-07 19:53:03,389 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646682483389","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:56848","multi.service_calls":0,"queuetimems":44433,"class":"HRegionServer","multi.mutations":1} 2022-03-07 19:53:09,932 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646681589930","responsesize":"23142","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 14 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.31.0.219:56742","multi.service_calls":0,"queuetimems":50978,"class":"HRegionServer","multi.mutations":14} 2022-03-07 19:53:11,324 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646681891323","responsesize":"14879","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 9 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:38572","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":9} 2022-03-07 19:53:36,238 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646682516237","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:38684","multi.service_calls":0,"queuetimems":27,"class":"HRegionServer","multi.mutations":1} 2022-03-07 19:53:38,173 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646682218173","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:37774","multi.service_calls":0,"queuetimems":34784,"class":"HRegionServer","multi.mutations":3} 2022-03-07 19:53:41,812 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646682221811","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:37774","multi.service_calls":0,"queuetimems":38422,"class":"HRegionServer","multi.mutations":3} 2022-03-07 19:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=391556, hits=383446, hitRatio=97.93%, , cachingAccesses=383576, cachingHits=383137, cachingHitsRatio=99.89%, evictions=23116, evicted=0, evictedPerRun=0.0 2022-03-07 19:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 19:57:18,958 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646682738957","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:38726","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 19:57:39,734 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646682759733","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:56880","multi.service_calls":0,"queuetimems":20775,"class":"HRegionServer","multi.mutations":1} 2022-03-07 19:58:03,391 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646682783390","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:38732","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 19:58:09,932 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646682789932","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:38732","multi.service_calls":0,"queuetimems":6542,"class":"HRegionServer","multi.mutations":2} 2022-03-07 19:58:11,325 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646682791325","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:56886","multi.service_calls":0,"queuetimems":7934,"class":"HRegionServer","multi.mutations":1} 2022-03-07 19:58:24,137 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646682204135","responsesize":"18183","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 11 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900002,"client":"172.30.41.118:38624","multi.service_calls":0,"queuetimems":20746,"class":"HRegionServer","multi.mutations":11} 2022-03-07 19:58:36,213 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646682516210","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600003,"client":"172.30.41.155:37792","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 19:58:36,238 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646682816238","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:56886","multi.service_calls":0,"queuetimems":32847,"class":"HRegionServer","multi.mutations":1} 2022-03-07 19:58:38,174 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646682818174","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:37808","multi.service_calls":0,"queuetimems":34784,"class":"HRegionServer","multi.mutations":1} 2022-03-07 19:58:41,813 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646682821812","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:56886","multi.service_calls":0,"queuetimems":38421,"class":"HRegionServer","multi.mutations":1} 2022-03-07 20:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=392345, hits=384235, hitRatio=97.93%, , cachingAccesses=384365, cachingHits=383926, cachingHitsRatio=99.89%, evictions=23146, evicted=0, evictedPerRun=0.0 2022-03-07 20:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 20:02:19,727 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646682739726","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:37806","multi.service_calls":0,"queuetimems":768,"class":"HRegionServer","multi.mutations":2} 2022-03-07 20:02:25,468 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646682445465","responsesize":"18183","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 11 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900003,"client":"172.30.41.118:38676","multi.service_calls":0,"queuetimems":6509,"class":"HRegionServer","multi.mutations":11} 2022-03-07 20:02:39,735 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646683059734","responsesize":"6613","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:56914","multi.service_calls":0,"queuetimems":20775,"class":"HRegionServer","multi.mutations":4} 2022-03-07 20:02:48,896 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646682768895","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 6 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:38726","multi.service_calls":0,"queuetimems":29938,"class":"HRegionServer","multi.mutations":6} 2022-03-07 20:03:03,391 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646683083391","responsesize":"6613","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:56914","multi.service_calls":0,"queuetimems":44432,"class":"HRegionServer","multi.mutations":4} 2022-03-07 20:03:24,140 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646683104138","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.118:38784","multi.service_calls":0,"queuetimems":14205,"class":"HRegionServer","multi.mutations":1} 2022-03-07 20:03:38,177 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646683118174","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300003,"client":"172.31.0.219:56920","multi.service_calls":0,"queuetimems":1955,"class":"HRegionServer","multi.mutations":1} 2022-03-07 20:03:41,811 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646682521810","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 5 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:37792","multi.service_calls":0,"queuetimems":5600,"class":"HRegionServer","multi.mutations":5} 2022-03-07 20:03:41,813 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646683121813","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300000,"client":"172.30.41.118:38784","multi.service_calls":0,"queuetimems":17675,"class":"HRegionServer","multi.mutations":1} 2022-03-07 20:03:44,135 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646682524133","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.31.0.219:56854","multi.service_calls":0,"queuetimems":7923,"class":"HRegionServer","multi.mutations":4} 2022-03-07 20:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=393103, hits=384993, hitRatio=97.94%, , cachingAccesses=385123, cachingHits=384684, cachingHitsRatio=99.89%, evictions=23176, evicted=0, evictedPerRun=0.0 2022-03-07 20:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 20:07:19,728 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646683339728","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:37842","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 20:07:25,469 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646683345469","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:37842","multi.service_calls":0,"queuetimems":5741,"class":"HRegionServer","multi.mutations":1} 2022-03-07 20:07:39,736 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646683359735","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:37842","multi.service_calls":0,"queuetimems":20007,"class":"HRegionServer","multi.mutations":1} 2022-03-07 20:07:48,896 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646683368896","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:37842","multi.service_calls":0,"queuetimems":29168,"class":"HRegionServer","multi.mutations":1} 2022-03-07 20:08:11,326 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646683091325","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:37826","multi.service_calls":0,"queuetimems":52366,"class":"HRegionServer","multi.mutations":5} 2022-03-07 20:08:36,220 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646683116219","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:37828","multi.service_calls":0,"queuetimems":4,"class":"HRegionServer","multi.mutations":2} 2022-03-07 20:08:36,244 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646683116238","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 8 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600006,"client":"172.30.41.118:38784","multi.service_calls":0,"queuetimems":26305,"class":"HRegionServer","multi.mutations":8} 2022-03-07 20:08:41,812 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646683421811","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:38870","multi.service_calls":0,"queuetimems":3632,"class":"HRegionServer","multi.mutations":1} 2022-03-07 20:08:41,814 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646683421814","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:37844","multi.service_calls":0,"queuetimems":3635,"class":"HRegionServer","multi.mutations":1} 2022-03-07 20:08:44,135 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646683424135","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:38870","multi.service_calls":0,"queuetimems":5955,"class":"HRegionServer","multi.mutations":1} 2022-03-07 20:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=393868, hits=385758, hitRatio=97.94%, , cachingAccesses=385888, cachingHits=385449, cachingHitsRatio=99.89%, evictions=23206, evicted=0, evictedPerRun=0.0 2022-03-07 20:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 20:12:18,960 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646683038959","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 5 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:37826","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":5} 2022-03-07 20:12:19,730 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646683639730","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:37860","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 20:12:25,471 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646683645470","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:38928","multi.service_calls":0,"queuetimems":5740,"class":"HRegionServer","multi.mutations":1} 2022-03-07 20:12:39,736 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646683659736","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:37860","multi.service_calls":0,"queuetimems":20006,"class":"HRegionServer","multi.mutations":1} 2022-03-07 20:12:48,897 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646683668896","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:56950","multi.service_calls":0,"queuetimems":29166,"class":"HRegionServer","multi.mutations":1} 2022-03-07 20:13:03,393 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646683383392","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 4 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:38852","multi.service_calls":0,"queuetimems":43664,"class":"HRegionServer","multi.mutations":4} 2022-03-07 20:13:36,222 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646683716222","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:37862","multi.service_calls":0,"queuetimems":24895,"class":"HRegionServer","multi.mutations":1} 2022-03-07 20:13:41,818 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646683721815","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300003,"client":"172.31.0.219:56952","multi.service_calls":0,"queuetimems":5593,"class":"HRegionServer","multi.mutations":1} 2022-03-07 20:13:51,411 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646682531408","responsesize":"19841","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 12 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1500002,"client":"172.30.41.118:38684","multi.service_calls":0,"queuetimems":15198,"class":"HRegionServer","multi.mutations":12} 2022-03-07 20:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=394569, hits=386459, hitRatio=97.94%, , cachingAccesses=386589, cachingHits=386150, cachingHitsRatio=99.89%, evictions=23236, evicted=0, evictedPerRun=0.0 2022-03-07 20:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 20:17:18,962 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646683938962","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:37876","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 20:17:39,737 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646683959737","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:37876","multi.service_calls":0,"queuetimems":20006,"class":"HRegionServer","multi.mutations":1} 2022-03-07 20:17:48,897 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646683968897","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:37876","multi.service_calls":0,"queuetimems":29935,"class":"HRegionServer","multi.mutations":2} 2022-03-07 20:18:09,934 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646683089933","responsesize":"11578","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 7 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200001,"client":"172.31.0.219:56914","multi.service_calls":0,"queuetimems":30198,"class":"HRegionServer","multi.mutations":7} 2022-03-07 20:18:11,328 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646683691327","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:38938","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":5} 2022-03-07 20:18:24,143 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646683404140","responsesize":"16531","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 10 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900002,"client":"172.30.41.118:38852","multi.service_calls":0,"queuetimems":64412,"class":"HRegionServer","multi.mutations":10} 2022-03-07 20:18:38,186 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646683418180","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900006,"client":"172.30.41.155:37844","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":7} 2022-03-07 20:18:41,815 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646683721812","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 5 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600003,"client":"172.30.41.155:37862","multi.service_calls":0,"queuetimems":30484,"class":"HRegionServer","multi.mutations":5} 2022-03-07 20:18:41,820 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646684021820","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:37878","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-07 20:18:44,140 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646683724136","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600004,"client":"172.31.0.219:56952","multi.service_calls":0,"queuetimems":32808,"class":"HRegionServer","multi.mutations":3} 2022-03-07 20:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=395252, hits=387142, hitRatio=97.95%, , cachingAccesses=387272, cachingHits=386833, cachingHitsRatio=99.89%, evictions=23266, evicted=0, evictedPerRun=0.0 2022-03-07 20:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 20:22:18,964 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646684238963","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:37892","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 20:22:25,472 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646683945472","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 5 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:38994","multi.service_calls":0,"queuetimems":6510,"class":"HRegionServer","multi.mutations":5} 2022-03-07 20:22:39,738 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646684259738","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:39068","multi.service_calls":0,"queuetimems":16993,"class":"HRegionServer","multi.mutations":1} 2022-03-07 20:22:48,898 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646684268898","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:39072","multi.service_calls":0,"queuetimems":29905,"class":"HRegionServer","multi.mutations":2} 2022-03-07 20:23:03,394 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646683983394","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:37878","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 20:23:09,935 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646684289935","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:56982","multi.service_calls":0,"queuetimems":44005,"class":"HRegionServer","multi.mutations":1} 2022-03-07 20:23:11,329 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646684291328","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:39068","multi.service_calls":0,"queuetimems":40036,"class":"HRegionServer","multi.mutations":1} 2022-03-07 20:23:24,144 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646684304144","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:37894","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-07 20:23:36,223 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646684016222","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:37878","multi.service_calls":0,"queuetimems":32828,"class":"HRegionServer","multi.mutations":2} 2022-03-07 20:23:36,250 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646683716244","responsesize":"19835","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 12 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900006,"client":"172.31.0.219:56952","multi.service_calls":0,"queuetimems":24916,"class":"HRegionServer","multi.mutations":12} 2022-03-07 20:23:41,815 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646684321815","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:39080","multi.service_calls":0,"queuetimems":17671,"class":"HRegionServer","multi.mutations":1} 2022-03-07 20:23:41,821 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646684321820","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:37894","multi.service_calls":0,"queuetimems":17676,"class":"HRegionServer","multi.mutations":2} 2022-03-07 20:23:44,141 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646684324140","responsesize":"6613","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:56984","multi.service_calls":0,"queuetimems":19996,"class":"HRegionServer","multi.mutations":4} 2022-03-07 20:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=396022, hits=387912, hitRatio=97.95%, , cachingAccesses=388042, cachingHits=387603, cachingHitsRatio=99.89%, evictions=23296, evicted=0, evictedPerRun=0.0 2022-03-07 20:26:48,063 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 20:28:09,936 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646684589935","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:57000","multi.service_calls":0,"queuetimems":50970,"class":"HRegionServer","multi.mutations":2} 2022-03-07 20:28:36,224 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646684616224","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:57002","multi.service_calls":0,"queuetimems":12078,"class":"HRegionServer","multi.mutations":1} 2022-03-07 20:28:36,251 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646684616251","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:57002","multi.service_calls":0,"queuetimems":27,"class":"HRegionServer","multi.mutations":1} 2022-03-07 20:28:38,187 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646684318186","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:39080","multi.service_calls":0,"queuetimems":14043,"class":"HRegionServer","multi.mutations":4} 2022-03-07 20:28:41,816 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646684621815","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:57002","multi.service_calls":0,"queuetimems":5591,"class":"HRegionServer","multi.mutations":2} 2022-03-07 20:28:51,412 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646684031411","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 6 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:37878","multi.service_calls":0,"queuetimems":9592,"class":"HRegionServer","multi.mutations":6} 2022-03-07 20:31:44,444 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-07 20:31:44,444 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=396708, hits=388598, hitRatio=97.96%, , cachingAccesses=388728, cachingHits=388289, cachingHitsRatio=99.89%, evictions=23326, evicted=0, evictedPerRun=0.0 2022-03-07 20:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 20:32:18,966 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646684538964","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:39136","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 20:32:19,733 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646683939731","responsesize":"23142","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 14 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200001,"client":"172.30.41.118:38994","multi.service_calls":0,"queuetimems":769,"class":"HRegionServer","multi.mutations":14} 2022-03-07 20:32:39,740 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646684559739","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:37908","multi.service_calls":0,"queuetimems":20774,"class":"HRegionServer","multi.mutations":3} 2022-03-07 20:32:48,899 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646684568898","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:39136","multi.service_calls":0,"queuetimems":9159,"class":"HRegionServer","multi.mutations":5} 2022-03-07 20:33:03,395 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646684583395","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 5 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:37908","multi.service_calls":0,"queuetimems":44430,"class":"HRegionServer","multi.mutations":5} 2022-03-07 20:33:09,937 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646684889937","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:39226","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 20:33:11,330 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646684591329","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:57000","multi.service_calls":0,"queuetimems":31590,"class":"HRegionServer","multi.mutations":2} 2022-03-07 20:33:36,225 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646684916224","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:39226","multi.service_calls":0,"queuetimems":26287,"class":"HRegionServer","multi.mutations":2} 2022-03-07 20:33:38,188 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646684918187","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:37930","multi.service_calls":0,"queuetimems":28250,"class":"HRegionServer","multi.mutations":1} 2022-03-07 20:33:41,817 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646684921816","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:37930","multi.service_calls":0,"queuetimems":31879,"class":"HRegionServer","multi.mutations":3} 2022-03-07 20:33:44,142 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646684624141","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:37910","multi.service_calls":0,"queuetimems":7890,"class":"HRegionServer","multi.mutations":2} 2022-03-07 20:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=397453, hits=389343, hitRatio=97.96%, , cachingAccesses=389473, cachingHits=389034, cachingHitsRatio=99.89%, evictions=23356, evicted=0, evictedPerRun=0.0 2022-03-07 20:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 20:37:18,967 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646685138967","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:39276","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 20:37:19,734 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646685139733","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:37946","multi.service_calls":0,"queuetimems":766,"class":"HRegionServer","multi.mutations":2} 2022-03-07 20:37:25,475 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646684545473","responsesize":"18183","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 11 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900002,"client":"172.30.41.118:39136","multi.service_calls":0,"queuetimems":6508,"class":"HRegionServer","multi.mutations":11} 2022-03-07 20:37:48,900 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646685168900","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:57030","multi.service_calls":0,"queuetimems":29933,"class":"HRegionServer","multi.mutations":1} 2022-03-07 20:38:03,397 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646685183396","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:39290","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 20:38:36,225 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646685216225","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:57032","multi.service_calls":0,"queuetimems":32828,"class":"HRegionServer","multi.mutations":1} 2022-03-07 20:38:38,188 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646685218188","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:39290","multi.service_calls":0,"queuetimems":26857,"class":"HRegionServer","multi.mutations":1} 2022-03-07 20:38:41,817 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646685221817","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:37948","multi.service_calls":0,"queuetimems":30486,"class":"HRegionServer","multi.mutations":1} 2022-03-07 20:38:41,822 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646684621821","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 5 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:37910","multi.service_calls":0,"queuetimems":5570,"class":"HRegionServer","multi.mutations":5} 2022-03-07 20:38:44,142 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646685224142","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:37948","multi.service_calls":0,"queuetimems":32811,"class":"HRegionServer","multi.mutations":2} 2022-03-07 20:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=398098, hits=389988, hitRatio=97.96%, , cachingAccesses=390118, cachingHits=389679, cachingHitsRatio=99.89%, evictions=23386, evicted=0, evictedPerRun=0.0 2022-03-07 20:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 20:42:19,735 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646685439734","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:57046","multi.service_calls":0,"queuetimems":766,"class":"HRegionServer","multi.mutations":1} 2022-03-07 20:42:25,477 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646685445475","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.31.0.219:57046","multi.service_calls":0,"queuetimems":6507,"class":"HRegionServer","multi.mutations":1} 2022-03-07 20:43:09,938 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646685189937","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 5 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:37948","multi.service_calls":0,"queuetimems":6541,"class":"HRegionServer","multi.mutations":5} 2022-03-07 20:43:11,331 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646685191330","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:37948","multi.service_calls":0,"queuetimems":7934,"class":"HRegionServer","multi.mutations":4} 2022-03-07 20:43:24,149 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646684604145","responsesize":"21490","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 13 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200004,"client":"172.31.0.219:57002","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":13} 2022-03-07 20:43:36,226 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646685516226","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.155:37516","multi.service_calls":0,"queuetimems":20327,"class":"HRegionServer","multi.mutations":2} 2022-03-07 20:43:38,189 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646685518189","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:39362","multi.service_calls":0,"queuetimems":1963,"class":"HRegionServer","multi.mutations":1} 2022-03-07 20:43:41,823 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646685521822","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:39362","multi.service_calls":0,"queuetimems":5596,"class":"HRegionServer","multi.mutations":1} 2022-03-07 20:43:51,413 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646684931412","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 8 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:57020","multi.service_calls":0,"queuetimems":41475,"class":"HRegionServer","multi.mutations":8} 2022-03-07 20:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=398873, hits=390763, hitRatio=97.97%, , cachingAccesses=390893, cachingHits=390454, cachingHitsRatio=99.89%, evictions=23416, evicted=0, evictedPerRun=0.0 2022-03-07 20:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 20:47:18,969 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646685438968","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:39354","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 20:47:19,736 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646685739736","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:39398","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 20:47:39,741 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646685159740","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 7 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:39276","multi.service_calls":0,"queuetimems":20773,"class":"HRegionServer","multi.mutations":7} 2022-03-07 20:48:09,939 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646685789939","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:39410","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 20:48:11,332 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646685791332","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:38004","multi.service_calls":0,"queuetimems":1393,"class":"HRegionServer","multi.mutations":3} 2022-03-07 20:48:24,150 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646685804149","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:39410","multi.service_calls":0,"queuetimems":14210,"class":"HRegionServer","multi.mutations":1} 2022-03-07 20:48:36,254 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646684916252","responsesize":"11578","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 7 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200002,"client":"172.31.0.219:57020","multi.service_calls":0,"queuetimems":26315,"class":"HRegionServer","multi.mutations":7} 2022-03-07 20:48:38,190 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646685818189","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 1 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:39410","multi.service_calls":0,"queuetimems":26857,"class":"HRegionServer","multi.mutations":1} 2022-03-07 20:48:41,823 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646685521822","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":600001,"client":"172.30.41.118:39362","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 20:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=399525, hits=391415, hitRatio=97.97%, , cachingAccesses=391545, cachingHits=391106, cachingHitsRatio=99.89%, evictions=23446, evicted=0, evictedPerRun=0.0 2022-03-07 20:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 20:52:18,971 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646686038970","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:39450","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 20:52:19,737 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646686039737","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:57082","multi.service_calls":0,"queuetimems":767,"class":"HRegionServer","multi.mutations":1} 2022-03-07 20:52:39,742 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646686059742","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.155:38032","multi.service_calls":0,"queuetimems":20742,"class":"HRegionServer","multi.mutations":2} 2022-03-07 20:52:48,901 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646685468900","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:37966","multi.service_calls":0,"queuetimems":29932,"class":"HRegionServer","multi.mutations":3} 2022-03-07 20:53:03,399 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646685483397","responsesize":"19835","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 12 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900002,"client":"172.30.41.118:39354","multi.service_calls":0,"queuetimems":44429,"class":"HRegionServer","multi.mutations":12} 2022-03-07 20:53:36,227 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646685816226","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:38004","multi.service_calls":0,"queuetimems":24894,"class":"HRegionServer","multi.mutations":2} 2022-03-07 20:53:36,254 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646686116254","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:57084","multi.service_calls":0,"queuetimems":26313,"class":"HRegionServer","multi.mutations":1} 2022-03-07 20:53:41,823 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646685821823","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:39410","multi.service_calls":0,"queuetimems":31884,"class":"HRegionServer","multi.mutations":2} 2022-03-07 20:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=400210, hits=392100, hitRatio=97.97%, , cachingAccesses=392230, cachingHits=391791, cachingHitsRatio=99.89%, evictions=23476, evicted=0, evictedPerRun=0.0 2022-03-07 20:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 20:57:18,972 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646686338971","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:39518","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 20:57:19,738 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646686339738","responsesize":"6613","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:38052","multi.service_calls":0,"queuetimems":766,"class":"HRegionServer","multi.mutations":4} 2022-03-07 20:57:25,480 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646685745477","responsesize":"14879","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 9 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900003,"client":"172.30.41.118:39398","multi.service_calls":0,"queuetimems":5741,"class":"HRegionServer","multi.mutations":9} 2022-03-07 20:57:39,743 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646686359742","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:38052","multi.service_calls":0,"queuetimems":20004,"class":"HRegionServer","multi.mutations":1} 2022-03-07 20:57:48,902 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646686368901","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:39518","multi.service_calls":0,"queuetimems":29929,"class":"HRegionServer","multi.mutations":1} 2022-03-07 20:58:09,941 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646686089940","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:38040","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 20:58:36,256 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646686416255","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:39528","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 20:58:38,191 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646686118190","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:39454","multi.service_calls":0,"queuetimems":28250,"class":"HRegionServer","multi.mutations":4} 2022-03-07 20:58:44,144 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646685524142","responsesize":"23142","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 14 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.118:39362","multi.service_calls":0,"queuetimems":2319,"class":"HRegionServer","multi.mutations":14} 2022-03-07 20:58:51,415 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646685831413","responsesize":"19835","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 12 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900002,"client":"172.30.41.118:39410","multi.service_calls":0,"queuetimems":41474,"class":"HRegionServer","multi.mutations":12} 2022-03-07 21:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=400849, hits=392739, hitRatio=97.98%, , cachingAccesses=392869, cachingHits=392430, cachingHitsRatio=99.89%, evictions=23506, evicted=0, evictedPerRun=0.0 2022-03-07 21:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 21:02:18,973 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646686638973","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:39572","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 21:02:25,480 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646686645480","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:39572","multi.service_calls":0,"queuetimems":6507,"class":"HRegionServer","multi.mutations":1} 2022-03-07 21:03:03,401 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646686383400","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:57102","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 21:03:09,943 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646686689942","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:39586","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 21:03:11,334 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646686091333","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 8 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:38040","multi.service_calls":0,"queuetimems":1393,"class":"HRegionServer","multi.mutations":8} 2022-03-07 21:03:36,257 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646686716256","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:39586","multi.service_calls":0,"queuetimems":26314,"class":"HRegionServer","multi.mutations":2} 2022-03-07 21:03:41,824 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646686121823","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 5 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:57084","multi.service_calls":0,"queuetimems":31882,"class":"HRegionServer","multi.mutations":5} 2022-03-07 21:03:44,145 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646686724144","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:38068","multi.service_calls":0,"queuetimems":34202,"class":"HRegionServer","multi.mutations":1} 2022-03-07 21:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=401635, hits=393525, hitRatio=97.98%, , cachingAccesses=393655, cachingHits=393216, cachingHitsRatio=99.89%, evictions=23536, evicted=0, evictedPerRun=0.0 2022-03-07 21:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 21:07:19,740 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646686639739","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:38066","multi.service_calls":0,"queuetimems":766,"class":"HRegionServer","multi.mutations":2} 2022-03-07 21:07:48,903 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646686668902","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:38066","multi.service_calls":0,"queuetimems":29929,"class":"HRegionServer","multi.mutations":4} 2022-03-07 21:08:09,973 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646686989972","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300001,"client":"172.30.41.118:39650","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 21:08:36,229 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646686416227","responsesize":"21487","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 13 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.31.0.219:57102","multi.service_calls":0,"queuetimems":32827,"class":"HRegionServer","multi.mutations":13} 2022-03-07 21:08:36,257 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646687016257","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:38082","multi.service_calls":0,"queuetimems":32855,"class":"HRegionServer","multi.mutations":1} 2022-03-07 21:08:41,825 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646686421824","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 6 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:38054","multi.service_calls":0,"queuetimems":5569,"class":"HRegionServer","multi.mutations":6} 2022-03-07 21:08:51,416 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646686731415","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:39586","multi.service_calls":0,"queuetimems":15158,"class":"HRegionServer","multi.mutations":2} 2022-03-07 21:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=402386, hits=394276, hitRatio=97.98%, , cachingAccesses=394406, cachingHits=393967, cachingHitsRatio=99.89%, evictions=23566, evicted=0, evictedPerRun=0.0 2022-03-07 21:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 21:12:18,975 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646686938974","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:39642","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":5} 2022-03-07 21:12:19,741 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646687239741","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:38098","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 21:12:25,482 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646686945480","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 4 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.155:38078","multi.service_calls":0,"queuetimems":6506,"class":"HRegionServer","multi.mutations":4} 2022-03-07 21:12:39,744 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646686659743","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 5 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:57116","multi.service_calls":0,"queuetimems":20770,"class":"HRegionServer","multi.mutations":5} 2022-03-07 21:12:48,904 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646687268903","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:39714","multi.service_calls":0,"queuetimems":29162,"class":"HRegionServer","multi.mutations":2} 2022-03-07 21:13:03,403 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646686983402","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:39642","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 21:13:24,154 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646686104150","responsesize":"24797","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 15 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1500004,"client":"172.30.41.118:39454","multi.service_calls":0,"queuetimems":14210,"class":"HRegionServer","multi.mutations":15} 2022-03-07 21:13:36,258 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646687316258","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:38100","multi.service_calls":0,"queuetimems":28,"class":"HRegionServer","multi.mutations":2} 2022-03-07 21:13:38,192 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646686718191","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 7 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:38068","multi.service_calls":0,"queuetimems":28249,"class":"HRegionServer","multi.mutations":7} 2022-03-07 21:13:41,825 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646687021825","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:38082","multi.service_calls":0,"queuetimems":30490,"class":"HRegionServer","multi.mutations":3} 2022-03-07 21:13:51,417 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646687331416","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:57148","multi.service_calls":0,"queuetimems":15186,"class":"HRegionServer","multi.mutations":1} 2022-03-07 21:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=403226, hits=395116, hitRatio=97.99%, , cachingAccesses=395246, cachingHits=394807, cachingHitsRatio=99.89%, evictions=23596, evicted=0, evictedPerRun=0.0 2022-03-07 21:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 21:17:18,976 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646687538976","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:39770","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 21:17:19,742 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646687539742","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:38124","multi.service_calls":0,"queuetimems":766,"class":"HRegionServer","multi.mutations":1} 2022-03-07 21:18:11,336 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646686991335","responsesize":"18183","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 11 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.30.41.118:39642","multi.service_calls":0,"queuetimems":7933,"class":"HRegionServer","multi.mutations":11} 2022-03-07 21:18:36,230 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646687316230","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:39728","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-07 21:18:36,259 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646687616258","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:38130","multi.service_calls":0,"queuetimems":32854,"class":"HRegionServer","multi.mutations":3} 2022-03-07 21:18:38,192 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646687618192","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:57166","multi.service_calls":0,"queuetimems":34788,"class":"HRegionServer","multi.mutations":3} 2022-03-07 21:18:41,826 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646687621825","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:39782","multi.service_calls":0,"queuetimems":17670,"class":"HRegionServer","multi.mutations":1} 2022-03-07 21:18:41,826 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646687321825","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:38100","multi.service_calls":0,"queuetimems":5595,"class":"HRegionServer","multi.mutations":2} 2022-03-07 21:18:51,418 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646687631417","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:38130","multi.service_calls":0,"queuetimems":48013,"class":"HRegionServer","multi.mutations":1} 2022-03-07 21:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=404014, hits=395904, hitRatio=97.99%, , cachingAccesses=396034, cachingHits=395595, cachingHitsRatio=99.89%, evictions=23626, evicted=0, evictedPerRun=0.0 2022-03-07 21:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 21:22:19,743 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646687839743","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:57182","multi.service_calls":0,"queuetimems":765,"class":"HRegionServer","multi.mutations":1} 2022-03-07 21:22:39,745 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646687559744","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 6 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:39770","multi.service_calls":0,"queuetimems":20002,"class":"HRegionServer","multi.mutations":6} 2022-03-07 21:22:48,905 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646687568904","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 4 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:38124","multi.service_calls":0,"queuetimems":29928,"class":"HRegionServer","multi.mutations":4} 2022-03-07 21:23:03,404 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646687583403","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 8 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:39782","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":8} 2022-03-07 21:23:24,156 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646687604154","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.118:39782","multi.service_calls":0,"queuetimems":20751,"class":"HRegionServer","multi.mutations":2} 2022-03-07 21:23:36,260 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646687916259","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:38168","multi.service_calls":0,"queuetimems":28,"class":"HRegionServer","multi.mutations":1} 2022-03-07 21:23:41,826 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646687921826","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:39838","multi.service_calls":0,"queuetimems":5595,"class":"HRegionServer","multi.mutations":1} 2022-03-07 21:23:44,146 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646687024145","responsesize":"13230","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 8 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200001,"client":"172.30.41.155:38082","multi.service_calls":0,"queuetimems":32810,"class":"HRegionServer","multi.mutations":8} 2022-03-07 21:24:15,666 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-07 21:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=404923, hits=396813, hitRatio=98.00%, , cachingAccesses=396943, cachingHits=396504, cachingHitsRatio=99.89%, evictions=23656, evicted=0, evictedPerRun=0.0 2022-03-07 21:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 21:27:18,978 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646687838977","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 5 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:39832","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":5} 2022-03-07 21:27:25,485 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646687545482","responsesize":"18183","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 11 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900003,"client":"172.30.41.118:39770","multi.service_calls":0,"queuetimems":6506,"class":"HRegionServer","multi.mutations":11} 2022-03-07 21:27:39,746 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646688159745","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:38198","multi.service_calls":0,"queuetimems":20001,"class":"HRegionServer","multi.mutations":2} 2022-03-07 21:28:09,975 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646687289973","responsesize":"18186","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 11 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.31.0.219:57146","multi.service_calls":0,"queuetimems":50231,"class":"HRegionServer","multi.mutations":11} 2022-03-07 21:28:24,157 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646688204156","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:39892","multi.service_calls":0,"queuetimems":35250,"class":"HRegionServer","multi.mutations":1} 2022-03-07 21:28:36,232 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646687916231","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:38168","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 21:28:36,261 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646688216260","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300000,"client":"172.30.41.118:39892","multi.service_calls":0,"queuetimems":15005,"class":"HRegionServer","multi.mutations":1} 2022-03-07 21:28:38,193 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646687918193","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.31.0.219:57184","multi.service_calls":0,"queuetimems":1961,"class":"HRegionServer","multi.mutations":5} 2022-03-07 21:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=405799, hits=397689, hitRatio=98.00%, , cachingAccesses=397819, cachingHits=397380, cachingHitsRatio=99.89%, evictions=23686, evicted=0, evictedPerRun=0.0 2022-03-07 21:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-07 21:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 21:32:18,980 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646688438980","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:38232","multi.service_calls":0,"queuetimems":1,"class":"HRegionServer","multi.mutations":1} 2022-03-07 21:32:19,745 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646688139744","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:39892","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 21:32:48,906 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646688168906","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 8 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:39892","multi.service_calls":0,"queuetimems":29162,"class":"HRegionServer","multi.mutations":8} 2022-03-07 21:33:11,963 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646688491962","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.30.41.118:40004","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 21:33:36,233 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646688516233","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:40006","multi.service_calls":0,"queuetimems":12075,"class":"HRegionServer","multi.mutations":1} 2022-03-07 21:33:38,194 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646688518194","responsesize":"6613","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:38238","multi.service_calls":0,"queuetimems":14036,"class":"HRegionServer","multi.mutations":4} 2022-03-07 21:33:41,827 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646687921826","responsesize":"18183","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 11 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.30.41.118:39838","multi.service_calls":0,"queuetimems":5594,"class":"HRegionServer","multi.mutations":11} 2022-03-07 21:33:41,830 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646688221827","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 6 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600003,"client":"172.31.0.219:57204","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":6} 2022-03-07 21:33:51,419 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646687931418","responsesize":"18183","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 11 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.31.0.219:57184","multi.service_calls":0,"queuetimems":15186,"class":"HRegionServer","multi.mutations":11} 2022-03-07 21:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=406617, hits=398507, hitRatio=98.01%, , cachingAccesses=398637, cachingHits=398198, cachingHitsRatio=99.89%, evictions=23716, evicted=0, evictedPerRun=0.0 2022-03-07 21:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 21:37:18,981 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646688738981","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:38262","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 21:37:19,746 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646688739746","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:38262","multi.service_calls":0,"queuetimems":765,"class":"HRegionServer","multi.mutations":1} 2022-03-07 21:37:39,748 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646688459747","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:39988","multi.service_calls":0,"queuetimems":20767,"class":"HRegionServer","multi.mutations":4} 2022-03-07 21:37:48,907 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646688768907","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:40060","multi.service_calls":0,"queuetimems":29926,"class":"HRegionServer","multi.mutations":1} 2022-03-07 21:38:03,406 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646688183404","responsesize":"16531","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 10 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900002,"client":"172.31.0.219:57200","multi.service_calls":0,"queuetimems":43659,"class":"HRegionServer","multi.mutations":10} 2022-03-07 21:38:11,395 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646687891336","responsesize":"18186","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 11 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200059,"client":"172.30.41.118:39832","multi.service_calls":0,"queuetimems":52358,"class":"HRegionServer","multi.mutations":11} 2022-03-07 21:38:11,964 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646688791964","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:38266","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 21:38:24,160 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646688504158","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 3 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.155:38238","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 21:38:36,234 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646688816233","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:40076","multi.service_calls":0,"queuetimems":24269,"class":"HRegionServer","multi.mutations":1} 2022-03-07 21:38:36,262 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646688516261","responsesize":"13224","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 8 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:40006","multi.service_calls":0,"queuetimems":12103,"class":"HRegionServer","multi.mutations":8} 2022-03-07 21:38:38,195 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646688818194","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:38266","multi.service_calls":0,"queuetimems":26230,"class":"HRegionServer","multi.mutations":1} 2022-03-07 21:38:41,828 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646688821827","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:38266","multi.service_calls":0,"queuetimems":29863,"class":"HRegionServer","multi.mutations":1} 2022-03-07 21:38:41,831 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646688821831","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:38266","multi.service_calls":0,"queuetimems":29867,"class":"HRegionServer","multi.mutations":1} 2022-03-07 21:38:44,148 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646688224146","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 8 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":900002,"client":"172.30.41.118:39892","multi.service_calls":0,"queuetimems":55240,"class":"HRegionServer","multi.mutations":8} 2022-03-07 21:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=407384, hits=399274, hitRatio=98.01%, , cachingAccesses=399404, cachingHits=398965, cachingHitsRatio=99.89%, evictions=23746, evicted=0, evictedPerRun=0.0 2022-03-07 21:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 21:42:25,488 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646688445485","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 8 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.155:38232","multi.service_calls":0,"queuetimems":6505,"class":"HRegionServer","multi.mutations":8} 2022-03-07 21:42:48,908 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646689068907","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:38294","multi.service_calls":0,"queuetimems":29925,"class":"HRegionServer","multi.mutations":1} 2022-03-07 21:43:36,235 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646689116234","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:40146","multi.service_calls":0,"queuetimems":12073,"class":"HRegionServer","multi.mutations":1} 2022-03-07 21:43:36,262 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646689116262","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:38296","multi.service_calls":0,"queuetimems":12100,"class":"HRegionServer","multi.mutations":2} 2022-03-07 21:43:41,828 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646689121828","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:40148","multi.service_calls":0,"queuetimems":17636,"class":"HRegionServer","multi.mutations":2} 2022-03-07 21:43:41,832 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646689121831","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:57256","multi.service_calls":0,"queuetimems":17669,"class":"HRegionServer","multi.mutations":2} 2022-03-07 21:43:51,420 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646688831419","responsesize":"14876","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 9 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:40076","multi.service_calls":0,"queuetimems":39455,"class":"HRegionServer","multi.mutations":9} 2022-03-07 21:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=408177, hits=400067, hitRatio=98.01%, , cachingAccesses=400197, cachingHits=399758, cachingHitsRatio=99.89%, evictions=23776, evicted=0, evictedPerRun=0.0 2022-03-07 21:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 21:47:18,983 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646689038982","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:40136","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":5} 2022-03-07 21:48:03,407 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646689083406","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:40136","multi.service_calls":0,"queuetimems":43659,"class":"HRegionServer","multi.mutations":3} 2022-03-07 21:48:11,965 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646689091964","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:38294","multi.service_calls":0,"queuetimems":52217,"class":"HRegionServer","multi.mutations":2} 2022-03-07 21:48:36,236 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646689416236","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300000,"client":"172.30.41.118:40212","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 21:48:36,263 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646689416263","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:38318","multi.service_calls":0,"queuetimems":27,"class":"HRegionServer","multi.mutations":1} 2022-03-07 21:48:41,832 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646689421832","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:38318","multi.service_calls":0,"queuetimems":5569,"class":"HRegionServer","multi.mutations":1} 2022-03-07 21:48:44,148 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646689124148","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 2 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:40146","multi.service_calls":0,"queuetimems":7913,"class":"HRegionServer","multi.mutations":2} 2022-03-07 21:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=408928, hits=400818, hitRatio=98.02%, , cachingAccesses=400948, cachingHits=400509, cachingHitsRatio=99.89%, evictions=23806, evicted=0, evictedPerRun=0.0 2022-03-07 21:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 21:52:18,985 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646689638984","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:40262","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 21:52:19,748 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646689039747","responsesize":"9923","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 6 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:38294","multi.service_calls":0,"queuetimems":764,"class":"HRegionServer","multi.mutations":6} 2022-03-07 21:52:25,490 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646689345489","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:40206","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 21:52:48,909 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646689368908","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:38314","multi.service_calls":0,"queuetimems":23419,"class":"HRegionServer","multi.mutations":4} 2022-03-07 21:53:11,397 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646689091395","responsesize":"16531","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 10 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900002,"client":"172.31.0.219:57254","multi.service_calls":0,"queuetimems":52412,"class":"HRegionServer","multi.mutations":10} 2022-03-07 21:53:24,165 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646689104162","responsesize":"11575","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 7 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900003,"client":"172.30.41.155:38296","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":7} 2022-03-07 21:53:36,264 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646689716263","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:38336","multi.service_calls":0,"queuetimems":32855,"class":"HRegionServer","multi.mutations":1} 2022-03-07 21:53:38,196 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646689118195","responsesize":"14879","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 9 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:40146","multi.service_calls":0,"queuetimems":14034,"class":"HRegionServer","multi.mutations":9} 2022-03-07 21:53:41,829 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646689421828","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 6 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:38318","multi.service_calls":0,"queuetimems":5592,"class":"HRegionServer","multi.mutations":6} 2022-03-07 21:53:41,833 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646689721832","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:38336","multi.service_calls":0,"queuetimems":38424,"class":"HRegionServer","multi.mutations":2} 2022-03-07 21:53:44,149 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646689724149","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:57318","multi.service_calls":0,"queuetimems":40741,"class":"HRegionServer","multi.mutations":1} 2022-03-07 21:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=409656, hits=401546, hitRatio=98.02%, , cachingAccesses=401676, cachingHits=401237, cachingHitsRatio=99.89%, evictions=23836, evicted=0, evictedPerRun=0.0 2022-03-07 21:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 21:57:39,749 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646689059748","responsesize":"8274","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 5 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200001,"client":"172.31.0.219:57254","multi.service_calls":0,"queuetimems":20765,"class":"HRegionServer","multi.mutations":5} 2022-03-07 21:57:48,909 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646689968909","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:40310","multi.service_calls":0,"queuetimems":29160,"class":"HRegionServer","multi.mutations":1} 2022-03-07 21:58:03,409 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646689683408","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":600001,"client":"172.30.41.118:40268","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-07 21:58:24,166 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646690004166","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:40324","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 21:58:36,237 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646689716236","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 5 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:57318","multi.service_calls":0,"queuetimems":32828,"class":"HRegionServer","multi.mutations":5} 2022-03-07 21:58:41,833 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646690021833","responsesize":"9917","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 6 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:40324","multi.service_calls":0,"queuetimems":17667,"class":"HRegionServer","multi.mutations":6} 2022-03-07 21:58:44,150 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646690024149","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:38358","multi.service_calls":0,"queuetimems":5952,"class":"HRegionServer","multi.mutations":1} 2022-03-07 21:58:51,422 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646689431420","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 4 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900002,"client":"172.30.41.155:38318","multi.service_calls":0,"queuetimems":15157,"class":"HRegionServer","multi.mutations":4} 2022-03-07 22:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=410428, hits=402318, hitRatio=98.02%, , cachingAccesses=402448, cachingHits=402009, cachingHitsRatio=99.89%, evictions=23866, evicted=0, evictedPerRun=0.0 2022-03-07 22:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 22:02:18,987 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646689938986","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:38356","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 22:02:19,749 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646689939748","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:40310","multi.service_calls":0,"queuetimems":762,"class":"HRegionServer","multi.mutations":2} 2022-03-07 22:02:25,492 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646689945490","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.155:38356","multi.service_calls":0,"queuetimems":6504,"class":"HRegionServer","multi.mutations":3} 2022-03-07 22:02:39,751 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646690259750","responsesize":"6613","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:40384","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-07 22:02:48,910 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646690268910","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:38374","multi.service_calls":0,"queuetimems":9160,"class":"HRegionServer","multi.mutations":1} 2022-03-07 22:03:03,409 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646690283409","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:40384","multi.service_calls":0,"queuetimems":23659,"class":"HRegionServer","multi.mutations":3} 2022-03-07 22:03:11,398 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646689991397","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:57352","multi.service_calls":0,"queuetimems":51644,"class":"HRegionServer","multi.mutations":2} 2022-03-07 22:03:38,197 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646690018196","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:57354","multi.service_calls":0,"queuetimems":14030,"class":"HRegionServer","multi.mutations":3} 2022-03-07 22:03:41,830 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646690021829","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:40324","multi.service_calls":0,"queuetimems":3632,"class":"HRegionServer","multi.mutations":3} 2022-03-07 22:03:41,834 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646690321834","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:38376","multi.service_calls":0,"queuetimems":5595,"class":"HRegionServer","multi.mutations":1} 2022-03-07 22:03:51,422 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646690331422","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:38376","multi.service_calls":0,"queuetimems":15183,"class":"HRegionServer","multi.mutations":1} 2022-03-07 22:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=411162, hits=403052, hitRatio=98.03%, , cachingAccesses=403182, cachingHits=402743, cachingHitsRatio=99.89%, evictions=23896, evicted=0, evictedPerRun=0.0 2022-03-07 22:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 22:07:19,019 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646690539018","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:40456","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 22:07:19,750 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646690539750","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:38392","multi.service_calls":0,"queuetimems":111,"class":"HRegionServer","multi.mutations":1} 2022-03-07 22:07:25,492 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646690545492","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300000,"client":"172.30.41.118:40448","multi.service_calls":0,"queuetimems":6450,"class":"HRegionServer","multi.mutations":1} 2022-03-07 22:07:39,751 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646690559751","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:38392","multi.service_calls":0,"queuetimems":20001,"class":"HRegionServer","multi.mutations":1} 2022-03-07 22:07:48,911 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646690568910","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":300001,"client":"172.30.41.118:40454","multi.service_calls":0,"queuetimems":29160,"class":"HRegionServer","multi.mutations":1} 2022-03-07 22:08:03,410 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646690583409","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:57386","multi.service_calls":0,"queuetimems":39101,"class":"HRegionServer","multi.mutations":1} 2022-03-07 22:08:11,398 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646690591398","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:40454","multi.service_calls":0,"queuetimems":47222,"class":"HRegionServer","multi.mutations":1} 2022-03-07 22:08:11,967 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646689691965","responsesize":"21490","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 13 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200002,"client":"172.30.41.118:40268","multi.service_calls":0,"queuetimems":8557,"class":"HRegionServer","multi.mutations":13} 2022-03-07 22:08:36,265 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646690016264","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:38358","multi.service_calls":0,"queuetimems":12098,"class":"HRegionServer","multi.mutations":3} 2022-03-07 22:08:38,199 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646690618198","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:40466","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 22:08:41,831 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646690621830","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:57388","multi.service_calls":0,"queuetimems":3632,"class":"HRegionServer","multi.mutations":1} 2022-03-07 22:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=411872, hits=403762, hitRatio=98.03%, , cachingAccesses=403892, cachingHits=403453, cachingHitsRatio=99.89%, evictions=23926, evicted=0, evictedPerRun=0.0 2022-03-07 22:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 22:12:19,020 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646690839019","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:40524","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 22:12:19,752 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646690839751","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:40524","multi.service_calls":0,"queuetimems":732,"class":"HRegionServer","multi.mutations":2} 2022-03-07 22:12:39,752 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646690859751","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:38410","multi.service_calls":0,"queuetimems":20731,"class":"HRegionServer","multi.mutations":3} 2022-03-07 22:13:11,968 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646690891967","responsesize":"6613","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:57404","multi.service_calls":0,"queuetimems":8556,"class":"HRegionServer","multi.mutations":4} 2022-03-07 22:13:24,169 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646690304166","responsesize":"13227","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 8 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900003,"client":"172.30.41.155:38374","multi.service_calls":0,"queuetimems":44416,"class":"HRegionServer","multi.mutations":8} 2022-03-07 22:13:36,240 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646690316238","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:40398","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 22:13:36,266 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646690916265","responsesize":"6613","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:57404","multi.service_calls":0,"queuetimems":32854,"class":"HRegionServer","multi.mutations":4} 2022-03-07 22:13:38,200 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646690918199","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:40542","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 22:13:41,831 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646690921831","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:38412","multi.service_calls":0,"queuetimems":3632,"class":"HRegionServer","multi.mutations":1} 2022-03-07 22:13:41,835 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646690621835","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 4 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:40466","multi.service_calls":0,"queuetimems":3637,"class":"HRegionServer","multi.mutations":4} 2022-03-07 22:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=412597, hits=404487, hitRatio=98.03%, , cachingAccesses=404617, cachingHits=404178, cachingHitsRatio=99.89%, evictions=23956, evicted=0, evictedPerRun=0.0 2022-03-07 22:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 22:17:19,021 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646691139021","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:40594","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 22:17:25,494 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646690845492","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.155:38410","multi.service_calls":0,"queuetimems":6473,"class":"HRegionServer","multi.mutations":3} 2022-03-07 22:17:39,753 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646691159752","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 3 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:40594","multi.service_calls":0,"queuetimems":20731,"class":"HRegionServer","multi.mutations":3} 2022-03-07 22:18:11,399 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646690891398","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:38410","multi.service_calls":0,"queuetimems":51647,"class":"HRegionServer","multi.mutations":4} 2022-03-07 22:18:11,969 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646691191968","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:38428","multi.service_calls":0,"queuetimems":52947,"class":"HRegionServer","multi.mutations":2} 2022-03-07 22:18:24,171 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646691204169","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300002,"client":"172.30.41.155:38428","multi.service_calls":0,"queuetimems":65147,"class":"HRegionServer","multi.mutations":1} 2022-03-07 22:18:36,240 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646691216240","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:38428","multi.service_calls":0,"queuetimems":76488,"class":"HRegionServer","multi.mutations":1} 2022-03-07 22:18:41,836 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646691221836","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300000,"client":"172.30.41.118:40610","multi.service_calls":0,"queuetimems":3606,"class":"HRegionServer","multi.mutations":2} 2022-03-07 22:18:44,151 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646690324150","responsesize":"23142","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 14 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":1200001,"client":"172.30.41.118:40398","multi.service_calls":0,"queuetimems":7912,"class":"HRegionServer","multi.mutations":14} 2022-03-07 22:18:51,424 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646690631422","responsesize":"8271","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 5 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.155:38394","multi.service_calls":0,"queuetimems":13224,"class":"HRegionServer","multi.mutations":5} 2022-03-07 22:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=413357, hits=405247, hitRatio=98.04%, , cachingAccesses=405377, cachingHits=404938, cachingHitsRatio=99.89%, evictions=23986, evicted=0, evictedPerRun=0.0 2022-03-07 22:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 22:22:19,023 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646691439022","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:40664","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 22:22:39,754 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646691459753","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 1 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:38446","multi.service_calls":0,"queuetimems":20731,"class":"HRegionServer","multi.mutations":1} 2022-03-07 22:22:48,913 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646690868911","responsesize":"18183","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 11 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900002,"client":"172.31.0.219:57402","multi.service_calls":0,"queuetimems":29891,"class":"HRegionServer","multi.mutations":11} 2022-03-07 22:23:03,412 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646690883411","responsesize":"4967","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.30.41.118:40530","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 22:23:11,400 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646691491399","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:40664","multi.service_calls":0,"queuetimems":31646,"class":"HRegionServer","multi.mutations":1} 2022-03-07 22:23:24,172 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646691504172","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:38446","multi.service_calls":0,"queuetimems":44418,"class":"HRegionServer","multi.mutations":1} 2022-03-07 22:23:36,242 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646691516241","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:38448","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 22:23:36,267 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646691216266","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 6 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:38428","multi.service_calls":0,"queuetimems":77244,"class":"HRegionServer","multi.mutations":6} 2022-03-07 22:23:41,832 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646691221831","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:38430","multi.service_calls":0,"queuetimems":3630,"class":"HRegionServer","multi.mutations":4} 2022-03-07 22:23:44,152 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646691524151","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:38448","multi.service_calls":0,"queuetimems":7910,"class":"HRegionServer","multi.mutations":2} 2022-03-07 22:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=414073, hits=405963, hitRatio=98.04%, , cachingAccesses=406093, cachingHits=405654, cachingHitsRatio=99.89%, evictions=24016, evicted=0, evictedPerRun=0.0 2022-03-07 22:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 22:27:19,753 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646691139752","responsesize":"18183","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 11 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.30.41.118:40594","multi.service_calls":0,"queuetimems":731,"class":"HRegionServer","multi.mutations":11} 2022-03-07 22:27:25,497 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646691445495","responsesize":"9920","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 6 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600002,"client":"172.30.41.155:38446","multi.service_calls":0,"queuetimems":6473,"class":"HRegionServer","multi.mutations":6} 2022-03-07 22:27:39,755 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646691759754","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:38462","multi.service_calls":0,"queuetimems":20730,"class":"HRegionServer","multi.mutations":3} 2022-03-07 22:28:03,412 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646691783412","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:38462","multi.service_calls":0,"queuetimems":44388,"class":"HRegionServer","multi.mutations":2} 2022-03-07 22:28:11,970 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646691491969","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 4 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:40664","multi.service_calls":0,"queuetimems":52947,"class":"HRegionServer","multi.mutations":4} 2022-03-07 22:28:24,173 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646691804172","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:40738","multi.service_calls":0,"queuetimems":44417,"class":"HRegionServer","multi.mutations":1} 2022-03-07 22:28:36,242 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646691816242","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 1 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.118:40738","multi.service_calls":0,"queuetimems":77218,"class":"HRegionServer","multi.mutations":1} 2022-03-07 22:28:38,202 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646691218201","responsesize":"19835","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 12 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Tur ","processingtimems":900001,"client":"172.30.41.118:40612","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":12} 2022-03-07 22:28:41,838 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646691521837","responsesize":"8268","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 5 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:57440","multi.service_calls":0,"queuetimems":5596,"class":"HRegionServer","multi.mutations":5} 2022-03-07 22:28:44,153 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646691824152","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:40738","multi.service_calls":0,"queuetimems":19686,"class":"HRegionServer","multi.mutations":1} 2022-03-07 22:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=414887, hits=406777, hitRatio=98.05%, , cachingAccesses=406907, cachingHits=406468, cachingHitsRatio=99.89%, evictions=24046, evicted=0, evictedPerRun=0.0 2022-03-07 22:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-07 22:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 22:32:19,025 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646691739024","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 2 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:38462","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 22:32:19,755 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646692039755","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.31.0.219:57476","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 22:32:48,914 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646691768913","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 4 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:40738","multi.service_calls":0,"queuetimems":29889,"class":"HRegionServer","multi.mutations":4} 2022-03-07 22:33:11,401 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646691791400","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.31.0.219:57458","multi.service_calls":0,"queuetimems":52376,"class":"HRegionServer","multi.mutations":2} 2022-03-07 22:33:36,268 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646691816267","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 3 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:38462","multi.service_calls":0,"queuetimems":56512,"class":"HRegionServer","multi.mutations":3} 2022-03-07 22:33:38,252 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646692118252","responsesize":"1657","method":"Multi","param":"region= __test-table_cvp328.sjc.aristanetworks.com__,,1646451115727.4333b489291b1f77671b8f2e78779a1e., for 1 action(s) and 1st row key=__status_key___","processingtimems":300000,"client":"172.30.41.118:40820","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 22:33:41,833 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646691821832","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 2 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:38462","multi.service_calls":0,"queuetimems":62077,"class":"HRegionServer","multi.mutations":2} 2022-03-07 22:33:56,823 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646692136822","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 2 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.31.0.219:57480","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 22:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=415602, hits=407492, hitRatio=98.05%, , cachingAccesses=407622, cachingHits=407183, cachingHitsRatio=99.89%, evictions=24076, evicted=0, evictedPerRun=0.0 2022-03-07 22:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 22:37:25,498 WARN [RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646692045497","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:38478","multi.service_calls":0,"queuetimems":5742,"class":"HRegionServer","multi.mutations":2} 2022-03-07 22:37:39,756 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646692059755","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 3 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:38478","multi.service_calls":0,"queuetimems":20000,"class":"HRegionServer","multi.mutations":3} 2022-03-07 22:37:48,915 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646692368914","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:38498","multi.service_calls":0,"queuetimems":29158,"class":"HRegionServer","multi.mutations":1} 2022-03-07 22:38:03,414 WARN [RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646692083413","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:40814","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 22:38:11,970 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646692091970","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x0C,1646451286050.ccdacf4b414cb8d1bc548efc77c663cb., for 4 action(s) and 1st row key=\\x0C\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.31.0.219:57478","multi.service_calls":0,"queuetimems":8556,"class":"HRegionServer","multi.mutations":4} 2022-03-07 22:38:24,175 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646692104173","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x9A\\xC4\\x07Devi ","processingtimems":600002,"client":"172.30.41.118:40814","multi.service_calls":0,"queuetimems":20760,"class":"HRegionServer","multi.mutations":3} 2022-03-07 22:38:36,243 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646692116242","responsesize":"3312","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.31.0.219:57478","multi.service_calls":0,"queuetimems":12068,"class":"HRegionServer","multi.mutations":2} 2022-03-07 22:38:36,270 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646692416269","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 1 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:38504","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":1} 2022-03-07 22:38:47,181 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646692427180","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00}\\x94\\xC4\\x03NTP\\xC4 ","processingtimems":300001,"client":"172.31.0.219:57350","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 22:38:51,425 WARN [RpcServer.default.FPBQ.Fifo.handler=11,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646691531424","responsesize":"11578","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 7 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":1200001,"client":"172.30.41.118:40684","multi.service_calls":0,"queuetimems":15183,"class":"HRegionServer","multi.mutations":7} 2022-03-07 22:38:56,824 WARN [RpcServer.default.FPBQ.Fifo.handler=10,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646692436823","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 1 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:38504","multi.service_calls":0,"queuetimems":6984,"class":"HRegionServer","multi.mutations":1} 2022-03-07 22:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=416375, hits=408265, hitRatio=98.05%, , cachingAccesses=408395, cachingHits=407956, cachingHitsRatio=99.89%, evictions=24106, evicted=0, evictedPerRun=0.0 2022-03-07 22:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 22:42:19,026 WARN [RpcServer.default.FPBQ.Fifo.handler=5,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646692339026","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 4 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.118:40884","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":4} 2022-03-07 22:42:19,758 WARN [RpcServer.default.FPBQ.Fifo.handler=8,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646692339756","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 4 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.155:38498","multi.service_calls":0,"queuetimems":730,"class":"HRegionServer","multi.mutations":4} 2022-03-07 22:42:48,915 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646692668915","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:38520","multi.service_calls":0,"queuetimems":23416,"class":"HRegionServer","multi.mutations":2} 2022-03-07 22:43:11,402 WARN [RpcServer.default.FPBQ.Fifo.handler=15,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646692391401","responsesize":"6616","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 4 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.31.0.219:57494","multi.service_calls":0,"queuetimems":51644,"class":"HRegionServer","multi.mutations":4} 2022-03-07 22:43:24,176 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646692704175","responsesize":"1657","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 1 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:38520","multi.service_calls":0,"queuetimems":44418,"class":"HRegionServer","multi.mutations":1} 2022-03-07 22:43:36,245 WARN [RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646692716244","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x01,1646451286050.7534b3b1812808d2d3c16342fbf347a5., for 2 action(s) and 1st row key=\\x01\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.118:40952","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":2} 2022-03-07 22:43:36,271 WARN [RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646692716270","responsesize":"3309","method":"Multi","param":"region= aeris_v2,\\x09,1646451286050.9b121c22e18472616debd0996ee2a4cb., for 2 action(s) and 1st row key=\\x09\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300001,"client":"172.30.41.155:38522","multi.service_calls":0,"queuetimems":26,"class":"HRegionServer","multi.mutations":2} 2022-03-07 22:43:38,254 WARN [RpcServer.default.FPBQ.Fifo.handler=14,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646692418253","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600001,"client":"172.30.41.118:40904","multi.service_calls":0,"queuetimems":0,"class":"HRegionServer","multi.mutations":3} 2022-03-07 22:43:41,839 WARN [RpcServer.default.FPBQ.Fifo.handler=3,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646692121838","responsesize":"6619","method":"Multi","param":"region= aeris_v2,\\x02,1646451286050.7d1b53259f1d8280a47f7704f98f2c9f., for 4 action(s) and 1st row key=\\x02\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":900001,"client":"172.31.0.219:57478","multi.service_calls":0,"queuetimems":38424,"class":"HRegionServer","multi.mutations":4} 2022-03-07 22:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=417160, hits=409050, hitRatio=98.06%, , cachingAccesses=409180, cachingHits=408741, cachingHitsRatio=99.89%, evictions=24136, evicted=0, evictedPerRun=0.0 2022-03-07 22:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 22:48:11,971 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646692691971","responsesize":"4964","method":"Multi","param":"region= aeris_v2,\\x08,1646451286050.0a764e51258807790269e46c55b99391., for 3 action(s) and 1st row key=\\x08\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":600000,"client":"172.30.41.155:38520","multi.service_calls":0,"queuetimems":32214,"class":"HRegionServer","multi.mutations":3} 2022-03-07 22:48:24,176 WARN [RpcServer.default.FPBQ.Fifo.handler=9,queue=1,port=16201] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$MultiRequest)","multi.gets":0,"starttimems":"1646693004176","responsesize":"4961","method":"Multi","param":"region= aeris_v2,\\x12,1646451286050.80a1ffe1dc2d8f258a6367d204dc64df., for 3 action(s) and 1st row key=\\x12\\xCFb\"\\xDAX\\x00\\x00\\x00\\x01\\x93\\xC4\\x08Turb ","processingtimems":300000,"client":"172.30.41.155:38540","multi.service_calls":0,"queuetimems":12773,"class":"HRegionServer","multi.mutations":3} 2022-03-07 22:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=417902, hits=409792, hitRatio=98.06%, , cachingAccesses=409922, cachingHits=409483, cachingHitsRatio=99.89%, evictions=24166, evicted=0, evictedPerRun=0.0 2022-03-07 22:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 22:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=418629, hits=410519, hitRatio=98.06%, , cachingAccesses=410649, cachingHits=410210, cachingHitsRatio=99.89%, evictions=24196, evicted=0, evictedPerRun=0.0 2022-03-07 22:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 23:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=419196, hits=411086, hitRatio=98.07%, , cachingAccesses=411216, cachingHits=410777, cachingHitsRatio=99.89%, evictions=24226, evicted=0, evictedPerRun=0.0 2022-03-07 23:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 23:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=419615, hits=411505, hitRatio=98.07%, , cachingAccesses=411635, cachingHits=411196, cachingHitsRatio=99.89%, evictions=24256, evicted=0, evictedPerRun=0.0 2022-03-07 23:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 23:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=419963, hits=411853, hitRatio=98.07%, , cachingAccesses=411983, cachingHits=411544, cachingHitsRatio=99.89%, evictions=24286, evicted=0, evictedPerRun=0.0 2022-03-07 23:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 23:12:06,791 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3611ms No GCs detected 2022-03-07 23:12:08,464 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1173ms No GCs detected 2022-03-07 23:12:19,469 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-07 23:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=420293, hits=412183, hitRatio=98.07%, , cachingAccesses=412313, cachingHits=411874, cachingHitsRatio=99.89%, evictions=24316, evicted=0, evictedPerRun=0.0 2022-03-07 23:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 23:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=420576, hits=412466, hitRatio=98.07%, , cachingAccesses=412596, cachingHits=412157, cachingHitsRatio=99.89%, evictions=24346, evicted=0, evictedPerRun=0.0 2022-03-07 23:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 23:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=420863, hits=412753, hitRatio=98.07%, , cachingAccesses=412883, cachingHits=412444, cachingHitsRatio=99.89%, evictions=24376, evicted=0, evictedPerRun=0.0 2022-03-07 23:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 23:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=421102, hits=412992, hitRatio=98.07%, , cachingAccesses=413122, cachingHits=412683, cachingHitsRatio=99.89%, evictions=24406, evicted=0, evictedPerRun=0.0 2022-03-07 23:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-07 23:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 23:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=421343, hits=413233, hitRatio=98.08%, , cachingAccesses=413363, cachingHits=412924, cachingHitsRatio=99.89%, evictions=24436, evicted=0, evictedPerRun=0.0 2022-03-07 23:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 23:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=421605, hits=413495, hitRatio=98.08%, , cachingAccesses=413625, cachingHits=413186, cachingHitsRatio=99.89%, evictions=24466, evicted=0, evictedPerRun=0.0 2022-03-07 23:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 23:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=421835, hits=413725, hitRatio=98.08%, , cachingAccesses=413855, cachingHits=413416, cachingHitsRatio=99.89%, evictions=24496, evicted=0, evictedPerRun=0.0 2022-03-07 23:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 23:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=422027, hits=413917, hitRatio=98.08%, , cachingAccesses=414047, cachingHits=413608, cachingHitsRatio=99.89%, evictions=24526, evicted=0, evictedPerRun=0.0 2022-03-07 23:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-07 23:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=422251, hits=414141, hitRatio=98.08%, , cachingAccesses=414271, cachingHits=413832, cachingHitsRatio=99.89%, evictions=24556, evicted=0, evictedPerRun=0.0 2022-03-07 23:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 00:01:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=422471, hits=414361, hitRatio=98.08%, , cachingAccesses=414491, cachingHits=414052, cachingHitsRatio=99.89%, evictions=24586, evicted=0, evictedPerRun=0.0 2022-03-08 00:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 00:06:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=422619, hits=414509, hitRatio=98.08%, , cachingAccesses=414639, cachingHits=414200, cachingHitsRatio=99.89%, evictions=24616, evicted=0, evictedPerRun=0.0 2022-03-08 00:06:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 00:11:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=422833, hits=414723, hitRatio=98.08%, , cachingAccesses=414853, cachingHits=414414, cachingHitsRatio=99.89%, evictions=24646, evicted=0, evictedPerRun=0.0 2022-03-08 00:11:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 00:13:30,988 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4149ms No GCs detected 2022-03-08 00:13:33,251 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1763ms No GCs detected 2022-03-08 00:13:54,480 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-08 00:16:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=423035, hits=414925, hitRatio=98.08%, , cachingAccesses=415055, cachingHits=414616, cachingHitsRatio=99.89%, evictions=24676, evicted=0, evictedPerRun=0.0 2022-03-08 00:16:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 00:21:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=423181, hits=415071, hitRatio=98.08%, , cachingAccesses=415201, cachingHits=414762, cachingHitsRatio=99.89%, evictions=24706, evicted=0, evictedPerRun=0.0 2022-03-08 00:21:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 00:26:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=423391, hits=415281, hitRatio=98.08%, , cachingAccesses=415411, cachingHits=414972, cachingHitsRatio=99.89%, evictions=24736, evicted=0, evictedPerRun=0.0 2022-03-08 00:26:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 00:31:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=423575, hits=415465, hitRatio=98.09%, , cachingAccesses=415595, cachingHits=415156, cachingHitsRatio=99.89%, evictions=24766, evicted=0, evictedPerRun=0.0 2022-03-08 00:31:44,383 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-08 00:31:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 00:36:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=423751, hits=415641, hitRatio=98.09%, , cachingAccesses=415771, cachingHits=415332, cachingHitsRatio=99.89%, evictions=24796, evicted=0, evictedPerRun=0.0 2022-03-08 00:36:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 00:41:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=423899, hits=415789, hitRatio=98.09%, , cachingAccesses=415919, cachingHits=415480, cachingHitsRatio=99.89%, evictions=24826, evicted=0, evictedPerRun=0.0 2022-03-08 00:41:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 00:46:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=424053, hits=415943, hitRatio=98.09%, , cachingAccesses=416073, cachingHits=415634, cachingHitsRatio=99.89%, evictions=24856, evicted=0, evictedPerRun=0.0 2022-03-08 00:46:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 00:51:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=424268, hits=416158, hitRatio=98.09%, , cachingAccesses=416288, cachingHits=415849, cachingHitsRatio=99.89%, evictions=24886, evicted=0, evictedPerRun=0.0 2022-03-08 00:51:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 00:56:44,381 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=424447, hits=416337, hitRatio=98.09%, , cachingAccesses=416467, cachingHits=416028, cachingHitsRatio=99.89%, evictions=24916, evicted=0, evictedPerRun=0.0 2022-03-08 00:56:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 01:00:46,574 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1304ms No GCs detected 2022-03-08 01:01:44,759 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=424627, hits=416517, hitRatio=98.09%, , cachingAccesses=416647, cachingHits=416208, cachingHitsRatio=99.89%, evictions=24946, evicted=0, evictedPerRun=0.0 2022-03-08 01:01:48,026 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 01:02:43,602 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1814ms No GCs detected 2022-03-08 01:02:58,594 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2916ms No GCs detected 2022-03-08 01:03:15,884 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1493ms No GCs detected 2022-03-08 01:03:27,441 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1054ms No GCs detected 2022-03-08 01:06:47,241 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=424812, hits=416702, hitRatio=98.09%, , cachingAccesses=416832, cachingHits=416393, cachingHitsRatio=99.89%, evictions=24976, evicted=0, evictedPerRun=0.0 2022-03-08 01:06:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 01:10:09,968 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1052ms No GCs detected 2022-03-08 01:11:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=424992, hits=416882, hitRatio=98.09%, , cachingAccesses=417012, cachingHits=416573, cachingHitsRatio=99.89%, evictions=25006, evicted=0, evictedPerRun=0.0 2022-03-08 01:11:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 01:13:36,235 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2575ms No GCs detected 2022-03-08 01:16:46,625 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=425149, hits=417039, hitRatio=98.09%, , cachingAccesses=417169, cachingHits=416730, cachingHitsRatio=99.89%, evictions=25036, evicted=0, evictedPerRun=0.0 2022-03-08 01:16:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 01:21:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=425321, hits=417211, hitRatio=98.09%, , cachingAccesses=417341, cachingHits=416902, cachingHitsRatio=99.89%, evictions=25066, evicted=0, evictedPerRun=0.0 2022-03-08 01:21:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 01:26:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=425499, hits=417389, hitRatio=98.09%, , cachingAccesses=417519, cachingHits=417080, cachingHitsRatio=99.89%, evictions=25096, evicted=0, evictedPerRun=0.0 2022-03-08 01:26:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 01:31:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=425661, hits=417551, hitRatio=98.09%, , cachingAccesses=417681, cachingHits=417242, cachingHitsRatio=99.89%, evictions=25125, evicted=0, evictedPerRun=0.0 2022-03-08 01:31:46,628 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-08 01:31:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 01:31:51,345 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-08 01:33:14,279 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 5294ms No GCs detected 2022-03-08 01:33:18,238 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2459ms No GCs detected 2022-03-08 01:33:22,256 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-08 01:33:53,004 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-08 01:35:23,980 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-08 01:36:46,625 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=425824, hits=417714, hitRatio=98.10%, , cachingAccesses=417844, cachingHits=417405, cachingHitsRatio=99.89%, evictions=25155, evicted=0, evictedPerRun=0.0 2022-03-08 01:36:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 01:37:28,916 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1487ms No GCs detected 2022-03-08 01:37:55,173 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-08 01:41:41,264 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 5975ms No GCs detected 2022-03-08 01:41:43,479 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1093ms No GCs detected 2022-03-08 01:41:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=425986, hits=417876, hitRatio=98.10%, , cachingAccesses=418006, cachingHits=417567, cachingHitsRatio=99.89%, evictions=25185, evicted=0, evictedPerRun=0.0 2022-03-08 01:41:50,662 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 01:42:16,199 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1457ms No GCs detected 2022-03-08 01:42:26,184 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 7384ms No GCs detected 2022-03-08 01:42:26,191 INFO [regionserver/cvp328:16201.Chore.1] hbase.ScheduledChore: Chore: MemstoreFlusherChore missed its start time 2022-03-08 01:42:26,191 INFO [regionserver/cvp328:16201.Chore.1] hbase.ScheduledChore: Chore: CompactionChecker missed its start time 2022-03-08 01:42:28,649 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1465ms No GCs detected 2022-03-08 01:42:30,429 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-08 01:42:32,345 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1788ms No GCs detected 2022-03-08 01:42:37,381 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1056ms No GCs detected 2022-03-08 01:43:04,667 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4663ms No GCs detected 2022-03-08 01:43:27,516 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 5276ms No GCs detected 2022-03-08 01:43:35,976 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1884ms No GCs detected 2022-03-08 01:43:45,041 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1415ms No GCs detected 2022-03-08 01:43:53,947 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1904ms No GCs detected 2022-03-08 01:44:07,142 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-08 01:44:16,575 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 8435ms No GCs detected 2022-03-08 01:44:19,315 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2233ms No GCs detected 2022-03-08 01:44:35,762 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1016ms No GCs detected 2022-03-08 01:44:38,501 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-08 01:44:38,514 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 509ms. 2022-03-08 01:44:39,025 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: tNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 2 failover attempts. Trying to failover after sleeping for 1784ms. 2022-03-08 01:45:00,595 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 6136ms No GCs detected 2022-03-08 01:45:07,622 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 6527ms No GCs detected 2022-03-08 01:45:29,292 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1360ms No GCs detected 2022-03-08 01:46:02,090 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2907ms No GCs detected 2022-03-08 01:46:13,870 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1352ms No GCs detected 2022-03-08 01:46:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=426142, hits=418032, hitRatio=98.10%, , cachingAccesses=418162, cachingHits=417723, cachingHitsRatio=99.90%, evictions=25213, evicted=0, evictedPerRun=0.0 2022-03-08 01:46:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 01:46:59,427 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1474ms No GCs detected 2022-03-08 01:47:01,748 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1320ms No GCs detected 2022-03-08 01:47:15,546 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-08 01:51:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=426300, hits=418190, hitRatio=98.10%, , cachingAccesses=418320, cachingHits=417881, cachingHitsRatio=99.90%, evictions=25243, evicted=0, evictedPerRun=0.0 2022-03-08 01:51:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 01:52:54,190 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1119ms No GCs detected 2022-03-08 01:53:30,101 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1595ms No GCs detected 2022-03-08 01:54:35,771 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1434ms No GCs detected 2022-03-08 01:54:40,615 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4344ms No GCs detected 2022-03-08 01:55:22,395 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-08 01:56:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=426451, hits=418341, hitRatio=98.10%, , cachingAccesses=418471, cachingHits=418032, cachingHitsRatio=99.90%, evictions=25273, evicted=0, evictedPerRun=0.0 2022-03-08 01:56:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 01:58:37,058 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1833ms No GCs detected 2022-03-08 02:00:20,207 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4099ms No GCs detected 2022-03-08 02:00:32,097 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2448ms No GCs detected 2022-03-08 02:00:41,003 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2349ms No GCs detected 2022-03-08 02:00:46,257 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2513ms No GCs detected 2022-03-08 02:01:13,389 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1751ms No GCs detected 2022-03-08 02:01:20,018 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1494ms No GCs detected 2022-03-08 02:01:33,739 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-08 02:01:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=426607, hits=418497, hitRatio=98.10%, , cachingAccesses=418627, cachingHits=418188, cachingHitsRatio=99.90%, evictions=25303, evicted=0, evictedPerRun=0.0 2022-03-08 02:01:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 02:02:25,918 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1861ms No GCs detected 2022-03-08 02:03:10,136 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2797ms No GCs detected 2022-03-08 02:03:13,708 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3071ms No GCs detected 2022-03-08 02:03:32,835 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 5313ms No GCs detected 2022-03-08 02:03:49,966 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1410ms No GCs detected 2022-03-08 02:04:23,259 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2628ms No GCs detected 2022-03-08 02:04:43,439 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-08 02:05:06,425 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2094ms No GCs detected 2022-03-08 02:05:13,838 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-08 02:06:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=426765, hits=418655, hitRatio=98.10%, , cachingAccesses=418785, cachingHits=418346, cachingHitsRatio=99.90%, evictions=25332, evicted=0, evictedPerRun=0.0 2022-03-08 02:06:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 02:09:45,360 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1009ms No GCs detected 2022-03-08 02:10:17,147 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-08 02:10:27,328 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1058ms No GCs detected 2022-03-08 02:10:49,419 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1045ms No GCs detected 2022-03-08 02:10:50,024 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-08 02:11:20,977 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-08 02:11:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=426916, hits=418806, hitRatio=98.10%, , cachingAccesses=418936, cachingHits=418497, cachingHitsRatio=99.90%, evictions=25362, evicted=0, evictedPerRun=0.0 2022-03-08 02:11:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 02:14:38,997 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4655ms No GCs detected 2022-03-08 02:14:45,615 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1017ms No GCs detected 2022-03-08 02:14:49,679 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1112ms No GCs detected 2022-03-08 02:16:32,631 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2166ms No GCs detected 2022-03-08 02:16:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=427047, hits=418937, hitRatio=98.10%, , cachingAccesses=419067, cachingHits=418628, cachingHitsRatio=99.90%, evictions=25392, evicted=0, evictedPerRun=0.0 2022-03-08 02:16:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 02:17:51,790 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1125ms No GCs detected 2022-03-08 02:18:41,104 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 6334ms No GCs detected 2022-03-08 02:19:00,064 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1005ms No GCs detected 2022-03-08 02:19:01,885 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1320ms No GCs detected 2022-03-08 02:19:30,094 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-08 02:21:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=427198, hits=419088, hitRatio=98.10%, , cachingAccesses=419218, cachingHits=418779, cachingHitsRatio=99.90%, evictions=25422, evicted=0, evictedPerRun=0.0 2022-03-08 02:21:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 02:24:26,623 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1346ms No GCs detected 2022-03-08 02:25:04,447 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-08 02:26:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=427355, hits=419245, hitRatio=98.10%, , cachingAccesses=419375, cachingHits=418936, cachingHitsRatio=99.90%, evictions=25452, evicted=0, evictedPerRun=0.0 2022-03-08 02:26:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 02:27:17,333 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2279ms No GCs detected 2022-03-08 02:27:45,998 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1825ms No GCs detected 2022-03-08 02:28:01,497 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1566ms No GCs detected 2022-03-08 02:28:18,920 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2415ms No GCs detected 2022-03-08 02:28:36,935 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-08 02:29:10,368 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-08 02:29:21,720 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 5070ms No GCs detected 2022-03-08 02:29:36,080 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1454ms No GCs detected 2022-03-08 02:30:21,676 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1890ms No GCs detected 2022-03-08 02:31:04,616 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2529ms GC pool 'G1 Young Generation' had collection(s): count=1 time=2982ms 2022-03-08 02:31:14,235 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1540ms No GCs detected 2022-03-08 02:31:23,247 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1387ms No GCs detected 2022-03-08 02:31:32,758 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2088ms GC pool 'G1 Young Generation' had collection(s): count=1 time=2173ms 2022-03-08 02:31:37,761 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-08 02:31:46,628 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-08 02:31:46,722 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=427538, hits=419428, hitRatio=98.10%, , cachingAccesses=419558, cachingHits=419119, cachingHitsRatio=99.90%, evictions=25482, evicted=0, evictedPerRun=0.0 2022-03-08 02:31:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 02:32:21,700 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2789ms No GCs detected 2022-03-08 02:32:33,828 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1123ms No GCs detected 2022-03-08 02:32:40,854 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-08 02:32:40,866 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 632ms. 2022-03-08 02:32:57,992 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2568ms No GCs detected 2022-03-08 02:33:36,836 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2424ms No GCs detected 2022-03-08 02:34:07,903 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1549ms No GCs detected 2022-03-08 02:36:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=427714, hits=419604, hitRatio=98.10%, , cachingAccesses=419734, cachingHits=419295, cachingHitsRatio=99.90%, evictions=25511, evicted=0, evictedPerRun=0.0 2022-03-08 02:36:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 02:40:59,082 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1615ms No GCs detected 2022-03-08 02:41:01,281 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1698ms No GCs detected 2022-03-08 02:41:23,235 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-08 02:41:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=427862, hits=419752, hitRatio=98.10%, , cachingAccesses=419882, cachingHits=419443, cachingHitsRatio=99.90%, evictions=25541, evicted=0, evictedPerRun=0.0 2022-03-08 02:41:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 02:42:40,359 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1954ms No GCs detected 2022-03-08 02:42:43,305 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1445ms No GCs detected 2022-03-08 02:46:46,625 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=427993, hits=419883, hitRatio=98.11%, , cachingAccesses=420013, cachingHits=419574, cachingHitsRatio=99.90%, evictions=25571, evicted=0, evictedPerRun=0.0 2022-03-08 02:46:50,272 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 02:47:17,380 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2707ms No GCs detected 2022-03-08 02:47:26,905 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1087ms No GCs detected 2022-03-08 02:47:28,918 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-08 02:47:41,133 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2224ms No GCs detected 2022-03-08 02:48:00,102 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-08 02:48:26,360 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1990ms No GCs detected 2022-03-08 02:49:07,424 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-08 02:51:12,776 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1085ms No GCs detected 2022-03-08 02:51:17,402 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-08 02:51:37,072 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2327ms No GCs detected 2022-03-08 02:51:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=428150, hits=420040, hitRatio=98.11%, , cachingAccesses=420170, cachingHits=419731, cachingHitsRatio=99.90%, evictions=25601, evicted=0, evictedPerRun=0.0 2022-03-08 02:51:48,365 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-08 02:51:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 02:52:12,446 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1645ms No GCs detected 2022-03-08 02:52:49,276 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-08 02:56:36,602 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1684ms No GCs detected 2022-03-08 02:56:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=428303, hits=420193, hitRatio=98.11%, , cachingAccesses=420323, cachingHits=419884, cachingHitsRatio=99.90%, evictions=25630, evicted=0, evictedPerRun=0.0 2022-03-08 02:56:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 02:57:00,609 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-08 03:00:01,936 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-08 03:01:37,125 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2360ms No GCs detected 2022-03-08 03:01:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=428470, hits=420360, hitRatio=98.11%, , cachingAccesses=420490, cachingHits=420051, cachingHitsRatio=99.90%, evictions=25660, evicted=0, evictedPerRun=0.0 2022-03-08 03:01:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 03:02:06,061 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1621ms No GCs detected 2022-03-08 03:02:08,447 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1885ms No GCs detected 2022-03-08 03:02:40,951 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 5879ms No GCs detected 2022-03-08 03:02:42,917 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1465ms No GCs detected 2022-03-08 03:02:42,918 INFO [regionserver/cvp328:16201.Chore.3] hbase.ScheduledChore: Chore: CompactionChecker missed its start time 2022-03-08 03:02:42,918 INFO [regionserver/cvp328:16201.Chore.1] hbase.ScheduledChore: Chore: MemstoreFlusherChore missed its start time 2022-03-08 03:04:25,756 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1169ms No GCs detected 2022-03-08 03:06:42,100 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1456ms No GCs detected 2022-03-08 03:06:47,299 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4698ms No GCs detected 2022-03-08 03:06:47,336 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=428599, hits=420489, hitRatio=98.11%, , cachingAccesses=420619, cachingHits=420180, cachingHitsRatio=99.90%, evictions=25690, evicted=0, evictedPerRun=0.0 2022-03-08 03:06:50,420 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 03:06:50,443 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-08 03:07:23,541 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-08 03:07:27,397 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3718ms No GCs detected 2022-03-08 03:07:28,663 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 817ms. 2022-03-08 03:07:59,854 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-08 03:11:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=428758, hits=420648, hitRatio=98.11%, , cachingAccesses=420778, cachingHits=420339, cachingHitsRatio=99.90%, evictions=25720, evicted=0, evictedPerRun=0.0 2022-03-08 03:11:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 03:16:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=428890, hits=420780, hitRatio=98.11%, , cachingAccesses=420910, cachingHits=420471, cachingHitsRatio=99.90%, evictions=25750, evicted=0, evictedPerRun=0.0 2022-03-08 03:16:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 03:19:32,001 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1183ms No GCs detected 2022-03-08 03:19:35,980 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-08 03:19:44,145 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1133ms No GCs detected 2022-03-08 03:20:18,126 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2095ms No GCs detected 2022-03-08 03:20:20,172 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1045ms No GCs detected 2022-03-08 03:20:32,834 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1415ms No GCs detected 2022-03-08 03:20:36,041 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2705ms No GCs detected 2022-03-08 03:21:16,127 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1705ms No GCs detected 2022-03-08 03:21:49,768 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1909ms No GCs detected 2022-03-08 03:21:51,759 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 03:21:51,749 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=429060, hits=420950, hitRatio=98.11%, , cachingAccesses=421080, cachingHits=420641, cachingHitsRatio=99.90%, evictions=25780, evicted=0, evictedPerRun=0.0 2022-03-08 03:21:58,295 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2600ms No GCs detected 2022-03-08 03:22:44,414 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1827ms No GCs detected 2022-03-08 03:26:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=429208, hits=421098, hitRatio=98.11%, , cachingAccesses=421228, cachingHits=420789, cachingHitsRatio=99.90%, evictions=25810, evicted=0, evictedPerRun=0.0 2022-03-08 03:26:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 03:27:02,446 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-08 03:27:06,687 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1370ms No GCs detected 2022-03-08 03:27:08,765 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1578ms No GCs detected 2022-03-08 03:27:08,859 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 407ms. 2022-03-08 03:27:09,307 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: tNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 2 failover attempts. Trying to failover after sleeping for 1419ms. 2022-03-08 03:27:10,728 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: tNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 3 failover attempts. Trying to failover after sleeping for 1312ms. 2022-03-08 03:27:17,116 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1329ms No GCs detected 2022-03-08 03:27:23,998 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1355ms No GCs detected 2022-03-08 03:31:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=429346, hits=421236, hitRatio=98.11%, , cachingAccesses=421366, cachingHits=420927, cachingHitsRatio=99.90%, evictions=25840, evicted=0, evictedPerRun=0.0 2022-03-08 03:31:46,628 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-08 03:31:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 03:36:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=429475, hits=421365, hitRatio=98.11%, , cachingAccesses=421495, cachingHits=421056, cachingHitsRatio=99.90%, evictions=25870, evicted=0, evictedPerRun=0.0 2022-03-08 03:36:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 03:41:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=429629, hits=421519, hitRatio=98.11%, , cachingAccesses=421649, cachingHits=421210, cachingHitsRatio=99.90%, evictions=25900, evicted=0, evictedPerRun=0.0 2022-03-08 03:41:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 03:45:19,072 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-08 03:46:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=429792, hits=421682, hitRatio=98.11%, , cachingAccesses=421812, cachingHits=421373, cachingHitsRatio=99.90%, evictions=25930, evicted=0, evictedPerRun=0.0 2022-03-08 03:46:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 03:49:39,412 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1705ms No GCs detected 2022-03-08 03:49:45,322 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 5410ms No GCs detected 2022-03-08 03:50:04,074 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1031ms No GCs detected 2022-03-08 03:50:12,580 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 6005ms No GCs detected 2022-03-08 03:50:22,974 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-08 03:50:23,976 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 1 failover attempts. Trying to failover after sleeping for 504ms. 2022-03-08 03:50:24,482 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 2 failover attempts. Trying to failover after sleeping for 908ms. 2022-03-08 03:51:11,083 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3525ms No GCs detected 2022-03-08 03:51:26,106 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-08 03:51:46,625 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=429930, hits=421820, hitRatio=98.11%, , cachingAccesses=421950, cachingHits=421511, cachingHitsRatio=99.90%, evictions=25959, evicted=0, evictedPerRun=0.0 2022-03-08 03:51:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 03:53:52,150 INFO [regionserver/cvp328:16201.Chore.1] hbase.ScheduledChore: Chore: CompactionChecker missed its start time 2022-03-08 03:53:52,150 INFO [regionserver/cvp328:16201.Chore.1] hbase.ScheduledChore: Chore: MemstoreFlusherChore missed its start time 2022-03-08 03:53:52,163 WARN [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 11038ms No GCs detected 2022-03-08 03:53:56,790 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-08 03:56:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=430046, hits=421936, hitRatio=98.11%, , cachingAccesses=422066, cachingHits=421627, cachingHitsRatio=99.90%, evictions=25989, evicted=0, evictedPerRun=0.0 2022-03-08 03:56:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 03:57:26,546 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1685ms No GCs detected 2022-03-08 03:57:28,538 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-08 03:57:29,635 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 596ms. 2022-03-08 04:01:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=430186, hits=422076, hitRatio=98.11%, , cachingAccesses=422206, cachingHits=421767, cachingHitsRatio=99.90%, evictions=26019, evicted=0, evictedPerRun=0.0 2022-03-08 04:01:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 04:02:56,380 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1320ms No GCs detected 2022-03-08 04:03:56,309 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1315ms No GCs detected 2022-03-08 04:04:03,310 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-08 04:06:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=430349, hits=422239, hitRatio=98.12%, , cachingAccesses=422369, cachingHits=421930, cachingHitsRatio=99.90%, evictions=26049, evicted=0, evictedPerRun=0.0 2022-03-08 04:06:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 04:09:29,511 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1483ms No GCs detected 2022-03-08 04:09:36,147 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-08 04:11:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=430487, hits=422377, hitRatio=98.12%, , cachingAccesses=422507, cachingHits=422068, cachingHitsRatio=99.90%, evictions=26079, evicted=0, evictedPerRun=0.0 2022-03-08 04:11:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 04:16:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=430600, hits=422490, hitRatio=98.12%, , cachingAccesses=422620, cachingHits=422181, cachingHitsRatio=99.90%, evictions=26109, evicted=0, evictedPerRun=0.0 2022-03-08 04:16:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 04:21:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=430739, hits=422629, hitRatio=98.12%, , cachingAccesses=422759, cachingHits=422320, cachingHitsRatio=99.90%, evictions=26139, evicted=0, evictedPerRun=0.0 2022-03-08 04:21:50,287 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 04:22:02,804 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2226ms No GCs detected 2022-03-08 04:22:54,604 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2509ms No GCs detected 2022-03-08 04:23:04,644 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 9539ms No GCs detected 2022-03-08 04:23:07,528 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1751ms No GCs detected 2022-03-08 04:23:12,026 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1063ms No GCs detected 2022-03-08 04:23:20,147 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1815ms No GCs detected 2022-03-08 04:23:30,926 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 8236ms No GCs detected 2022-03-08 04:23:43,902 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-08 04:24:03,307 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1120ms No GCs detected 2022-03-08 04:26:17,651 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-08 04:26:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=430895, hits=422785, hitRatio=98.12%, , cachingAccesses=422915, cachingHits=422476, cachingHitsRatio=99.90%, evictions=26168, evicted=0, evictedPerRun=0.0 2022-03-08 04:26:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 04:31:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=431016, hits=422906, hitRatio=98.12%, , cachingAccesses=423036, cachingHits=422597, cachingHitsRatio=99.90%, evictions=26198, evicted=0, evictedPerRun=0.0 2022-03-08 04:31:46,628 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-08 04:31:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 04:32:46,676 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3470ms No GCs detected 2022-03-08 04:33:32,320 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 7279ms No GCs detected 2022-03-08 04:35:40,298 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1058ms No GCs detected 2022-03-08 04:36:07,243 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1642ms No GCs detected 2022-03-08 04:36:25,668 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-08 04:36:46,633 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=431147, hits=423037, hitRatio=98.12%, , cachingAccesses=423167, cachingHits=422728, cachingHitsRatio=99.90%, evictions=26228, evicted=0, evictedPerRun=0.0 2022-03-08 04:36:50,272 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 04:37:27,703 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-08 04:37:29,727 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 1 failover attempts. Trying to failover after sleeping for 873ms. 2022-03-08 04:37:31,103 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: tNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 2 failover attempts. Trying to failover after sleeping for 1286ms. 2022-03-08 04:41:08,358 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3794ms No GCs detected 2022-03-08 04:41:12,810 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2885ms No GCs detected 2022-03-08 04:41:19,268 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1349ms No GCs detected 2022-03-08 04:41:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=431286, hits=423176, hitRatio=98.12%, , cachingAccesses=423306, cachingHits=422867, cachingHitsRatio=99.90%, evictions=26258, evicted=0, evictedPerRun=0.0 2022-03-08 04:41:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 04:44:17,230 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-08 04:44:56,500 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1197ms No GCs detected 2022-03-08 04:46:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=431413, hits=423303, hitRatio=98.12%, , cachingAccesses=423433, cachingHits=422994, cachingHitsRatio=99.90%, evictions=26288, evicted=0, evictedPerRun=0.0 2022-03-08 04:46:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 04:47:25,682 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1263ms No GCs detected 2022-03-08 04:51:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=431544, hits=423434, hitRatio=98.12%, , cachingAccesses=423564, cachingHits=423125, cachingHitsRatio=99.90%, evictions=26317, evicted=0, evictedPerRun=0.0 2022-03-08 04:51:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 04:54:17,277 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1059ms No GCs detected 2022-03-08 04:54:50,996 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-08 04:55:00,346 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2456ms No GCs detected 2022-03-08 04:55:21,461 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1602ms No GCs detected 2022-03-08 04:56:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=431678, hits=423568, hitRatio=98.12%, , cachingAccesses=423698, cachingHits=423259, cachingHitsRatio=99.90%, evictions=26347, evicted=0, evictedPerRun=0.0 2022-03-08 04:56:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 05:00:02,962 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1874ms No GCs detected 2022-03-08 05:01:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=431816, hits=423706, hitRatio=98.12%, , cachingAccesses=423836, cachingHits=423397, cachingHitsRatio=99.90%, evictions=26377, evicted=0, evictedPerRun=0.0 2022-03-08 05:01:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 05:01:55,850 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-08 05:01:55,852 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 541ms. 2022-03-08 05:01:56,396 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: tNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 2 failover attempts. Trying to failover after sleeping for 1113ms. 2022-03-08 05:03:49,486 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1093ms No GCs detected 2022-03-08 05:03:58,250 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-08 05:05:20,336 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1732ms No GCs detected 2022-03-08 05:06:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=431930, hits=423820, hitRatio=98.12%, , cachingAccesses=423950, cachingHits=423511, cachingHitsRatio=99.90%, evictions=26407, evicted=0, evictedPerRun=0.0 2022-03-08 05:06:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 05:07:31,270 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-08 05:11:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=432063, hits=423953, hitRatio=98.12%, , cachingAccesses=424083, cachingHits=423644, cachingHitsRatio=99.90%, evictions=26437, evicted=0, evictedPerRun=0.0 2022-03-08 05:11:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 05:15:35,722 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-08 05:16:26,010 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1084ms No GCs detected 2022-03-08 05:16:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=432201, hits=424091, hitRatio=98.12%, , cachingAccesses=424221, cachingHits=423782, cachingHitsRatio=99.90%, evictions=26467, evicted=0, evictedPerRun=0.0 2022-03-08 05:16:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 05:21:25,655 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2692ms No GCs detected 2022-03-08 05:21:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=432314, hits=424204, hitRatio=98.12%, , cachingAccesses=424334, cachingHits=423895, cachingHitsRatio=99.90%, evictions=26497, evicted=0, evictedPerRun=0.0 2022-03-08 05:21:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 05:26:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=432447, hits=424337, hitRatio=98.12%, , cachingAccesses=424467, cachingHits=424028, cachingHitsRatio=99.90%, evictions=26527, evicted=0, evictedPerRun=0.0 2022-03-08 05:26:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 05:27:41,433 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-08 05:27:41,465 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 731ms. 2022-03-08 05:28:02,405 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4210ms No GCs detected 2022-03-08 05:28:04,279 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1373ms No GCs detected 2022-03-08 05:28:13,240 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-08 05:28:13,243 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 349ms. 2022-03-08 05:28:13,597 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: tNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 2 failover attempts. Trying to failover after sleeping for 1781ms. 2022-03-08 05:31:19,617 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1065ms No GCs detected 2022-03-08 05:31:44,525 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2317ms No GCs detected 2022-03-08 05:31:46,628 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-08 05:31:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 05:31:57,001 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 9475ms No GCs detected 2022-03-08 05:31:57,029 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=432579, hits=424469, hitRatio=98.13%, , cachingAccesses=424599, cachingHits=424160, cachingHitsRatio=99.90%, evictions=26557, evicted=0, evictedPerRun=0.0 2022-03-08 05:32:04,530 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 5257ms No GCs detected 2022-03-08 05:32:36,153 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: java.net.ConnectException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp328.sjc.aristanetworks.com:9001 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-08 05:36:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=432699, hits=424589, hitRatio=98.13%, , cachingAccesses=424719, cachingHits=424280, cachingHitsRatio=99.90%, evictions=26586, evicted=0, evictedPerRun=0.0 2022-03-08 05:36:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 05:41:27,354 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2432ms No GCs detected 2022-03-08 05:41:31,784 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1429ms No GCs detected 2022-03-08 05:41:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=432831, hits=424721, hitRatio=98.13%, , cachingAccesses=424851, cachingHits=424412, cachingHitsRatio=99.90%, evictions=26615, evicted=0, evictedPerRun=0.0 2022-03-08 05:41:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 05:46:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=432941, hits=424831, hitRatio=98.13%, , cachingAccesses=424961, cachingHits=424522, cachingHitsRatio=99.90%, evictions=26645, evicted=0, evictedPerRun=0.0 2022-03-08 05:46:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 05:51:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=433078, hits=424968, hitRatio=98.13%, , cachingAccesses=425098, cachingHits=424659, cachingHitsRatio=99.90%, evictions=26675, evicted=0, evictedPerRun=0.0 2022-03-08 05:51:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 05:52:02,828 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1581ms No GCs detected 2022-03-08 05:52:05,530 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2201ms No GCs detected 2022-03-08 05:56:16,164 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1413ms No GCs detected 2022-03-08 05:56:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=433199, hits=425089, hitRatio=98.13%, , cachingAccesses=425219, cachingHits=424780, cachingHitsRatio=99.90%, evictions=26705, evicted=0, evictedPerRun=0.0 2022-03-08 05:56:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 05:57:26,229 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-08 05:57:26,230 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: java.net.ConnectException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp328.sjc.aristanetworks.com:9001 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 814ms. 2022-03-08 05:58:46,321 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1467ms No GCs detected 2022-03-08 06:01:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=433346, hits=425236, hitRatio=98.13%, , cachingAccesses=425366, cachingHits=424927, cachingHitsRatio=99.90%, evictions=26735, evicted=0, evictedPerRun=0.0 2022-03-08 06:01:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 06:02:29,461 INFO [regionserver/cvp328:16201.Chore.3] hbase.ScheduledChore: Chore: MemstoreFlusherChore missed its start time 2022-03-08 06:02:29,463 WARN [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 15227ms No GCs detected 2022-03-08 06:02:29,461 INFO [regionserver/cvp328:16201.Chore.1] hbase.ScheduledChore: Chore: CompactionChecker missed its start time 2022-03-08 06:02:59,642 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1196ms No GCs detected 2022-03-08 06:03:30,804 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-08 06:04:34,264 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2600ms No GCs detected 2022-03-08 06:04:55,639 INFO [regionserver/cvp328:16201.Chore.3] hbase.ScheduledChore: Chore: MemstoreFlusherChore missed its start time 2022-03-08 06:04:55,640 INFO [regionserver/cvp328:16201.Chore.3] hbase.ScheduledChore: Chore: CompactionChecker missed its start time 2022-03-08 06:04:56,039 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4685ms No GCs detected 2022-03-08 06:05:04,180 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1313ms No GCs detected 2022-03-08 06:05:06,348 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1667ms No GCs detected 2022-03-08 06:05:20,079 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-08 06:05:24,102 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1746ms No GCs detected 2022-03-08 06:05:54,283 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-08 06:06:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=433511, hits=425401, hitRatio=98.13%, , cachingAccesses=425531, cachingHits=425092, cachingHitsRatio=99.90%, evictions=26765, evicted=0, evictedPerRun=0.0 2022-03-08 06:06:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 06:08:32,081 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1911ms No GCs detected 2022-03-08 06:08:54,772 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-08 06:11:26,791 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-08 06:11:33,238 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 390ms. 2022-03-08 06:11:33,965 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: tNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 2 failover attempts. Trying to failover after sleeping for 1408ms. 2022-03-08 06:11:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=433667, hits=425557, hitRatio=98.13%, , cachingAccesses=425687, cachingHits=425248, cachingHitsRatio=99.90%, evictions=26794, evicted=0, evictedPerRun=0.0 2022-03-08 06:11:50,735 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 06:11:53,009 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1777ms No GCs detected 2022-03-08 06:15:07,754 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-08 06:16:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=433786, hits=425676, hitRatio=98.13%, , cachingAccesses=425806, cachingHits=425367, cachingHitsRatio=99.90%, evictions=26824, evicted=0, evictedPerRun=0.0 2022-03-08 06:16:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 06:18:40,748 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-08 06:18:42,211 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2025ms No GCs detected 2022-03-08 06:18:42,222 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 380ms. 2022-03-08 06:18:42,616 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: tNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 2 failover attempts. Trying to failover after sleeping for 1655ms. 2022-03-08 06:21:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=433908, hits=425798, hitRatio=98.13%, , cachingAccesses=425928, cachingHits=425489, cachingHitsRatio=99.90%, evictions=26854, evicted=0, evictedPerRun=0.0 2022-03-08 06:21:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 06:23:32,953 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1127ms No GCs detected 2022-03-08 06:23:46,207 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-08 06:25:27,815 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1722ms No GCs detected 2022-03-08 06:25:57,296 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1097ms No GCs detected 2022-03-08 06:26:00,926 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1397ms No GCs detected 2022-03-08 06:26:02,471 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1045ms No GCs detected 2022-03-08 06:26:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=434032, hits=425922, hitRatio=98.13%, , cachingAccesses=426052, cachingHits=425613, cachingHitsRatio=99.90%, evictions=26884, evicted=0, evictedPerRun=0.0 2022-03-08 06:26:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 06:27:20,176 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1337ms No GCs detected 2022-03-08 06:27:21,214 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-08 06:27:23,109 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2432ms No GCs detected 2022-03-08 06:27:23,462 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 426ms. 2022-03-08 06:27:24,946 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 2 failover attempts. Trying to failover after sleeping for 1488ms. 2022-03-08 06:27:26,440 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: tNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 3 failover attempts. Trying to failover after sleeping for 3379ms. 2022-03-08 06:27:29,822 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: tNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 4 failover attempts. Trying to failover after sleeping for 5273ms. 2022-03-08 06:28:22,219 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 5319ms No GCs detected 2022-03-08 06:28:32,636 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1915ms No GCs detected 2022-03-08 06:28:35,298 INFO [regionserver/cvp328:16201.Chore.2] hbase.ScheduledChore: Chore: MemstoreFlusherChore missed its start time 2022-03-08 06:28:35,298 INFO [regionserver/cvp328:16201.Chore.2] hbase.ScheduledChore: Chore: CompactionChecker missed its start time 2022-03-08 06:28:47,308 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2578ms No GCs detected 2022-03-08 06:28:58,137 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3979ms No GCs detected 2022-03-08 06:29:06,693 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-08 06:29:19,503 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2444ms No GCs detected 2022-03-08 06:29:38,888 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-08 06:29:43,782 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1784ms No GCs detected 2022-03-08 06:29:45,818 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1535ms No GCs detected 2022-03-08 06:30:13,951 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1499ms No GCs detected 2022-03-08 06:30:26,699 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3106ms No GCs detected 2022-03-08 06:31:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=434157, hits=426047, hitRatio=98.13%, , cachingAccesses=426177, cachingHits=425738, cachingHitsRatio=99.90%, evictions=26913, evicted=0, evictedPerRun=0.0 2022-03-08 06:31:46,628 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-08 06:31:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 06:32:44,688 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2112ms No GCs detected 2022-03-08 06:32:47,965 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2777ms No GCs detected 2022-03-08 06:32:55,589 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1990ms No GCs detected 2022-03-08 06:34:09,754 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1376ms No GCs detected 2022-03-08 06:35:03,554 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2702ms No GCs detected 2022-03-08 06:35:26,471 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1040ms No GCs detected 2022-03-08 06:35:36,801 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-08 06:36:46,931 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=434271, hits=426161, hitRatio=98.13%, , cachingAccesses=426291, cachingHits=425852, cachingHitsRatio=99.90%, evictions=26943, evicted=0, evictedPerRun=0.0 2022-03-08 06:36:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 06:37:09,512 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-08 06:37:12,265 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 747ms. 2022-03-08 06:37:24,705 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1300ms No GCs detected 2022-03-08 06:37:45,653 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-08 06:38:45,790 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3651ms No GCs detected 2022-03-08 06:38:49,110 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2820ms No GCs detected 2022-03-08 06:39:04,939 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1352ms No GCs detected 2022-03-08 06:39:15,640 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4697ms No GCs detected 2022-03-08 06:39:20,709 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-08 06:39:20,723 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 1 failover attempts. Trying to failover after sleeping for 669ms. 2022-03-08 06:39:56,307 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1590ms No GCs detected 2022-03-08 06:39:58,672 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1865ms No GCs detected 2022-03-08 06:40:06,245 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-08 06:41:07,874 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-08 06:41:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=434387, hits=426277, hitRatio=98.13%, , cachingAccesses=426407, cachingHits=425968, cachingHitsRatio=99.90%, evictions=26972, evicted=0, evictedPerRun=0.0 2022-03-08 06:41:50,529 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 06:44:32,268 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2557ms No GCs detected 2022-03-08 06:45:07,283 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3054ms No GCs detected 2022-03-08 06:46:08,474 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1590ms No GCs detected 2022-03-08 06:46:14,192 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2217ms No GCs detected 2022-03-08 06:46:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=434505, hits=426395, hitRatio=98.13%, , cachingAccesses=426525, cachingHits=426086, cachingHitsRatio=99.90%, evictions=27002, evicted=0, evictedPerRun=0.0 2022-03-08 06:46:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 06:47:22,074 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-08 06:47:35,994 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1155ms No GCs detected 2022-03-08 06:51:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=434617, hits=426507, hitRatio=98.13%, , cachingAccesses=426637, cachingHits=426198, cachingHitsRatio=99.90%, evictions=27032, evicted=0, evictedPerRun=0.0 2022-03-08 06:51:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 06:53:17,859 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1699ms No GCs detected 2022-03-08 06:53:24,407 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1235ms No GCs detected 2022-03-08 06:53:53,322 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1034ms No GCs detected 2022-03-08 06:54:19,256 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1835ms No GCs detected 2022-03-08 06:56:46,562 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1286ms No GCs detected 2022-03-08 06:56:46,625 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=434736, hits=426626, hitRatio=98.13%, , cachingAccesses=426756, cachingHits=426317, cachingHitsRatio=99.90%, evictions=27062, evicted=0, evictedPerRun=0.0 2022-03-08 06:56:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 07:01:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=434853, hits=426743, hitRatio=98.14%, , cachingAccesses=426873, cachingHits=426434, cachingHitsRatio=99.90%, evictions=27092, evicted=0, evictedPerRun=0.0 2022-03-08 07:01:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 07:06:46,625 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=434975, hits=426865, hitRatio=98.14%, , cachingAccesses=426995, cachingHits=426556, cachingHitsRatio=99.90%, evictions=27122, evicted=0, evictedPerRun=0.0 2022-03-08 07:06:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 07:11:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=435080, hits=426970, hitRatio=98.14%, , cachingAccesses=427100, cachingHits=426661, cachingHitsRatio=99.90%, evictions=27152, evicted=0, evictedPerRun=0.0 2022-03-08 07:11:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 07:12:11,113 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1138ms No GCs detected 2022-03-08 07:16:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=435191, hits=427081, hitRatio=98.14%, , cachingAccesses=427211, cachingHits=426772, cachingHitsRatio=99.90%, evictions=27182, evicted=0, evictedPerRun=0.0 2022-03-08 07:16:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 07:21:46,625 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=435313, hits=427203, hitRatio=98.14%, , cachingAccesses=427333, cachingHits=426894, cachingHitsRatio=99.90%, evictions=27212, evicted=0, evictedPerRun=0.0 2022-03-08 07:21:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 07:26:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=435418, hits=427308, hitRatio=98.14%, , cachingAccesses=427438, cachingHits=426999, cachingHitsRatio=99.90%, evictions=27242, evicted=0, evictedPerRun=0.0 2022-03-08 07:26:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 07:29:31,575 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1044ms No GCs detected 2022-03-08 07:29:39,763 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 7688ms No GCs detected 2022-03-08 07:31:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=435537, hits=427427, hitRatio=98.14%, , cachingAccesses=427557, cachingHits=427118, cachingHitsRatio=99.90%, evictions=27271, evicted=0, evictedPerRun=0.0 2022-03-08 07:31:46,628 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-08 07:31:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 07:36:53,293 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=435652, hits=427542, hitRatio=98.14%, , cachingAccesses=427672, cachingHits=427233, cachingHitsRatio=99.90%, evictions=27301, evicted=0, evictedPerRun=0.0 2022-03-08 07:36:53,293 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 07:36:53,311 INFO [regionserver/cvp328:16201.Chore.2] hbase.ScheduledChore: Chore: MemstoreFlusherChore missed its start time 2022-03-08 07:36:53,311 INFO [regionserver/cvp328:16201.Chore.2] hbase.ScheduledChore: Chore: CompactionChecker missed its start time 2022-03-08 07:36:53,485 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 8123ms No GCs detected 2022-03-08 07:41:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=435775, hits=427665, hitRatio=98.14%, , cachingAccesses=427795, cachingHits=427356, cachingHitsRatio=99.90%, evictions=27331, evicted=0, evictedPerRun=0.0 2022-03-08 07:41:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 07:42:30,754 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-08 07:42:30,791 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 761ms. 2022-03-08 07:42:32,553 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 2 failover attempts. Trying to failover after sleeping for 1458ms. 2022-03-08 07:42:35,590 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1342ms No GCs detected 2022-03-08 07:42:35,764 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: tNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 3 failover attempts. Trying to failover after sleeping for 1760ms. 2022-03-08 07:42:57,829 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 6256ms No GCs detected 2022-03-08 07:43:08,891 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-08 07:43:39,306 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-08 07:46:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=435881, hits=427771, hitRatio=98.14%, , cachingAccesses=427901, cachingHits=427462, cachingHitsRatio=99.90%, evictions=27360, evicted=0, evictedPerRun=0.0 2022-03-08 07:46:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 07:51:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=435997, hits=427887, hitRatio=98.14%, , cachingAccesses=428017, cachingHits=427578, cachingHitsRatio=99.90%, evictions=27390, evicted=0, evictedPerRun=0.0 2022-03-08 07:51:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 07:56:46,625 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=436100, hits=427990, hitRatio=98.14%, , cachingAccesses=428120, cachingHits=427681, cachingHitsRatio=99.90%, evictions=27420, evicted=0, evictedPerRun=0.0 2022-03-08 07:56:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 07:59:24,867 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2180ms No GCs detected 2022-03-08 07:59:31,616 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1247ms No GCs detected 2022-03-08 07:59:47,432 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1546ms No GCs detected 2022-03-08 08:00:18,100 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-08 08:01:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=436210, hits=428100, hitRatio=98.14%, , cachingAccesses=428230, cachingHits=427791, cachingHitsRatio=99.90%, evictions=27450, evicted=0, evictedPerRun=0.0 2022-03-08 08:01:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 08:03:26,247 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 5566ms No GCs detected 2022-03-08 08:03:50,461 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-08 08:06:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=436324, hits=428214, hitRatio=98.14%, , cachingAccesses=428344, cachingHits=427905, cachingHitsRatio=99.90%, evictions=27480, evicted=0, evictedPerRun=0.0 2022-03-08 08:06:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 08:08:41,380 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3269ms No GCs detected 2022-03-08 08:08:43,219 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1338ms No GCs detected 2022-03-08 08:08:53,092 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-08 08:11:30,662 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1454ms No GCs detected 2022-03-08 08:11:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=436436, hits=428326, hitRatio=98.14%, , cachingAccesses=428456, cachingHits=428017, cachingHitsRatio=99.90%, evictions=27510, evicted=0, evictedPerRun=0.0 2022-03-08 08:11:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 08:11:53,895 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001. Trying to failover immediately. 2022-03-08 08:12:49,221 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 6027ms No GCs detected 2022-03-08 08:16:46,625 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=436545, hits=428435, hitRatio=98.14%, , cachingAccesses=428565, cachingHits=428126, cachingHitsRatio=99.90%, evictions=27539, evicted=0, evictedPerRun=0.0 2022-03-08 08:16:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 08:20:31,498 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 3419ms No GCs detected 2022-03-08 08:20:31,523 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.net.ConnectTimeoutException: Call From cvp328.sjc.aristanetworks.com/172.30.41.118 to cvp365.sjc.aristanetworks.com:9001 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 1000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=cvp365.sjc.aristanetworks.com/172.30.41.155:9001]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-08 08:20:32,401 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 571ms. 2022-03-08 08:21:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=436650, hits=428540, hitRatio=98.14%, , cachingAccesses=428670, cachingHits=428231, cachingHitsRatio=99.90%, evictions=27569, evicted=0, evictedPerRun=0.0 2022-03-08 08:21:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 08:23:03,574 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1684ms No GCs detected 2022-03-08 08:23:05,623 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1548ms No GCs detected 2022-03-08 08:23:12,373 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 6249ms No GCs detected 2022-03-08 08:23:16,021 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-08 08:23:16,059 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: ntNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 1 failover attempts. Trying to failover after sleeping for 725ms. 2022-03-08 08:23:16,797 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: tNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001 after 2 failover attempts. Trying to failover after sleeping for 1667ms. 2022-03-08 08:23:18,466 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: tNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp328.sjc.aristanetworks.com/172.30.41.118:9001 after 3 failover attempts. Trying to failover after sleeping for 2677ms. 2022-03-08 08:26:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=436772, hits=428662, hitRatio=98.14%, , cachingAccesses=428792, cachingHits=428353, cachingHitsRatio=99.90%, evictions=27599, evicted=0, evictedPerRun=0.0 2022-03-08 08:26:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 08:31:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=436880, hits=428770, hitRatio=98.14%, , cachingAccesses=428900, cachingHits=428461, cachingHitsRatio=99.90%, evictions=27629, evicted=0, evictedPerRun=0.0 2022-03-08 08:31:46,628 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2022-03-08 08:31:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 08:33:03,291 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1413ms No GCs detected 2022-03-08 08:33:42,002 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 8104ms No GCs detected 2022-03-08 08:33:42,056 INFO [regionserver/cvp328:16201.Chore.2] hbase.ScheduledChore: Chore: MemstoreFlusherChore missed its start time 2022-03-08 08:33:42,056 INFO [regionserver/cvp328:16201.Chore.2] hbase.ScheduledChore: Chore: CompactionChecker missed its start time 2022-03-08 08:33:54,572 INFO [LeaseRenewer:cvp@mycluster] retry.RetryInvocationHandler: org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:744) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943) , while invoking ClientNamenodeProtocolTranslatorPB.renewLease over cvp365.sjc.aristanetworks.com/172.30.41.155:9001. Trying to failover immediately. 2022-03-08 08:36:46,626 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=436992, hits=428882, hitRatio=98.14%, , cachingAccesses=429012, cachingHits=428573, cachingHitsRatio=99.90%, evictions=27659, evicted=0, evictedPerRun=0.0 2022-03-08 08:36:50,271 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B 2022-03-08 08:41:46,625 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=874.05 KB, freeSize=306.35 MB, max=307.20 MB, blockCount=34, accesses=437099, hits=428989, hitRatio=98.14%, , cachingAccesses=429119, cachingHits=428680, cachingHitsRatio=99.90%, evictions=27689, evicted=0, evictedPerRun=0.0 2022-03-08 08:41:50,272 INFO [cvp328:16201Replication Statistics #0] regionserver.Replication: Global stats: WAL Edits Buffer Used=0B, Limit=268435456B